Modern DS Software Development

Team Structure

• Developers
• Planning: Agile (SCRUM, Kanban)
• Coding: version control system (svn, git), integrated development environments (Eclipse, PyCharm)
• Building: makefile, Maven (pom.xml)
• Testing: Selenium, HP QTP
• DevOps: Jenkins
• Operations
• Deploy: Docker
• Operate
• Monitor: Nagios

Web Development

• browser: Firefox, Chrome, Safari
• PL: what to do (Java, JavaScript, Python, Ruby, PHP)
• HTML: what to show
• CSS: how to show
• database: how data is stored (MySQL, Oracle, MongoDB)
• data format: how data is exchanged (JSON, XML)
• framework: platform or content-management system (Struts/Java, Meteor/JavaScript, Django/Python, Rails/Ruby, WordPress/PHP, Drupal/PHP)
• front-end: browser, HTML, CSS, PL
• back-end: server, database
• client: user of a web application
• server: where application is stored
• protocol: how machines communicate (HTTP)
• cloud computing services
• compute power, database storage, DBMS, AI services;
• SaaS (Google Apps); PaaS (Amazon retail, Flipkart retail); IaaS (Google Drive, Amazon Web Services, Microsoft Azure);Network-as-a-Service or NaaS (Asianet ISP); Data-as-a-Service or DaaS

ELK Stack: ElasticSearch (full-text search engine, run ML); Logstash; Kibana (data analysis and visualization)

Databases

• Relational (SQL) databases
• have to have a schema
• poor horizontal scaling (possibly due to tight coupling between tables)
• e.g. SQL Server, MySQL
• NoSQL
• enables better horizontal scaling (adding new servers, partitions)
• no need to have a schema (good for constantly evolving data)
• flexible
• key-value storage (JSON)
• e.g. MongoDB, Cassandra
  

Docker

• facilitate faster application delivery and deployment; ensure portability by separating application from infrastructure; enables a microservice-based architecture
• wrap OS, dependencies, configuration, directories and files (and not data) into a single lightweight application
• follows a client-server architecture; relies on Linux virtualization, cgroups, namespaces
• use Dockerfile to build a Docker image
• FROM (choose base image), MAINTAINER (for documentation purpose), RUN (execute commands during image building), CMD (execute commands by the running image), ADD (files to image), ENV (define environment variables), VOLUME, WORKDIR (change working directory), USER
• save images on (public/private) registry, e.g. Docker Hub
• run images as a Docker container
• use volumes for persistent data
• Docker commands
  > docker pull, push
  > docker create, run, start, kill, rm, stop, commit
  > docker images, rmi, ps, logs

Kubernetes

• container orchestration platform
• automatically deploy, manage and scale containerized applications
• k8s cluster services
• application configuration (app1.yaml file)