Dev & Ops

My personal state of art

Context:

Solving problems.

Problem:

How to organize systems, in a simple maner to create and keep?

Architectural style

Microservices

Martin Fowler puts it best:

A component is a unit of software that is independently replaceable and upgradeable.

Their Primary way of componentizing (...) software is by breaking it down into services.

Services are independently deployable.

Services are out-of-process components who communicate with a mechanism such as a web service request.

Brings to the architecture:

Polyglot programming

Polyglot persistence

Best tool for each job

Pressure on the Ops:

Automation

Monitoring

Provisioning

Fast delivery

Problem:

How to make this services communicate with each other and the world?

Archtectural style

REST

REpresentational
State
Transfer

a.k.a: WEB

Resumindo

From Roy Fielding:

Client-Server
Cache(able)
Stateless
Uniform Interface
Layered Systems
~~Code on-Demand~~

In practice: HTTP

Make good use of the headers
Stateless
Cacheable (E-Tags)
URI as identifier
Verbs as actions (GET, POST, PUT...)
Formats (content-types):
- HTML
- JSON
- XML
- CSV

Problem:

How to work with the team?

And garantee code sync
And not lose code
And have access to old code
And so on...

GIT

Solves

Who has the last version?
Who changed this line?
How do I know what changed?
How to send code?
I want to try this differently...
And more.

Problem:

Which language to use?

Many!

Remember: Polyglot programming and use the best tool for each job.

PHP

The language of the web

Pros

I can code in PHP
You can code in PHP
Lots of resources
Wordpress, Drupal, Symfony, Facebook...
Lately has been shaping up good

Cons

Not all the community is mature
Easy to make a mess
Require discipline

Best now a days:

Namespaces, classes and PDO
Composer
PHPUnit or SimpleTest
Behat (BDD)
Components from Symfony ou Zend
Lavarel or Slim
Templates mustache
PHPTheRightWay.com

Ruby

The new language of the Web!

ok, nodejs is the new-new, but really...

Pros

I can code in ruby
Lot of new programmers can
Really easy to learn
Truly Object Oriented
Great for the web
Very popular and with modern tools
And is ok for ops tasks and scripts

Cons

Sometimes lacks standart on the community
Easy to make a mess
Lot's on decision on webserver
Easy to kill performance(but not that bad really)

How I use it

2.1
Bundler
RSpec
Sinatra
Mustache
net & json native

Clojure

Functional programming on the JVM

Prós

JVM and java interop.
LISP
Imutability == predictability
Code is data, data is code
Parallelism and concurrency
Java performance, if needed.
Great standart and great community

Cons

Learning curve
Hard to find programmers
JVM size and startup

How I use it

lein
midje
ring
compojure
immutant
core.async
prismatic libs
clj-http + data.json

Problem:

Where to persist data?

CouchDB

It is

REST
Schemaless
Documet-based
Changes Feed
Incremental
Efficient HTTP-based replication
Most trustable DB I have used
CON: hard/limited to query (map/reduce)

ElasticSearch

You know, for search!

It is

REST
Schemaless
Document-based
Efficient and flexible searching of anything
Easy to send data in
CON: Not your primary database

PostgreSQL

The great SQL DB.

It is

Relational: everybody can handle it
Lots of tools
PostGIS!
hstore/jsonstore
Foreign Data Wrappers
If going relational, go PostgreSQL

SQLite

Default simple SQL DB

It is

Relational: everybody can handle it
Faster them it seems
Lots of read, few parallels writes
Single file DB, no depency and simple OPS
WAL, Fulltext-search, SpatialIndex...

Problem:

How to make sure it all works?

TDD

We all know this.

TDD

~~We all knwo this.~~

BDD

Most of us knows: behavior tests.

Feature: Open and edit profiles

    Scenario: create profile
        Given I am on "/specie/Aphelandra longiflora"
        Then I login as "Diogo", "diogo@cncflora.net", "admin,analyst"
        And I press "create-btn"

    Scenario: Edit a profile, changes apply and metadata
        Given I am on "/specie/Aphelandra longiflora"
        When I login as "Bruno", "bruno@cncflora.net", "analyst", "ACANTHACEAE"
        And I follow "Editar"
        Then I should see "Contribuidor(es): [Bruno] ; Diogo"
        Then I should see "Taxonomic Notes"
        Then I fill field "textarea[id*='-taxonomicNotes-notes']" with "Hello, notes."
        And I press "Salvar"
        Then I should see "Status: open"
        Then I should see "Notas Taxonômicas"
        Then I should see "Hello, notes."

Problem:

My dev-env will become a mess!

or: it works on my machine

or: someone has to configure the new guy pc

Vagrant!

Development environments made easy.

Standart and automation of dev env.

Single configuration file (Vagrantfile) for each project
The dev env is part of the code now
Source control
Publish, share and replicate
Standart

Vagrantfile

Vagrant.configure("2") do |config|
  config.vm.box = "ubuntu/trusty32"

  config.vm.network "private_network", ip: "192.168.50.188"

  config.vm.provision :shell, :inline => "apt-get update && apt-get install openjdk-7-jdk curl git tmux vim htop redis-server -y"
  config.vm.provision :shell, :inline => "wget https://raw.github.com/technomancy/leiningen/stable/bin/lein -O /usr/bin/lein"
  config.vm.provision :shell, :inline => "chmod +x /usr/bin/lein"
end

That is just ruby...

Vagrantfile

Vagrant.configure("2") do |config|
  config.vm.box = "ubuntu/trusty64"

  config.vm.network "private_network", ip: "192.168.50.10"
  config.vm.network :forwarded_port, host: 8888, guest: 80, auto_correct: true

  config.vm.provider "virtualbox" do |v|
    v.memory = 2048
    v.cpus = 2
  end

  config.vm.provision "docker" do |d|
    d.run "coreos/etcd", name: "etcd", args: "-p 8001:80 -p 4001:4001"
  end

  config.vm.provision "chef_solo" do |chef|
    chef.add_recipe "apache"
    chef.add_recipe "couchdb"
  end
end

But really, docker-compose is better for this.

Problem:

My production will be a mess!

This actually is the dev env now

Docker

Build, ship and run!

It is

Single Dockerfile by project
Dev and prod is the same, and part of the code
Dev and Prod gets version controlled, automated and testable
Fast

How?

You will automate, but:


# At your machine
docker build -t cncflora/profiles . # Build this project image
docker push cncflora/profiles # Send it to repository
# On the server
docker pull cncflora/profiles # pull lastest image, incrementally
docker run -d -p 8080 -t cncflora/profiles # Runs it

Any server, same thing:

AWS
Google compute engine
VPS
Vmware, xen
openstack, openshift, cloudfoundry
Anything Linux

Dockerfile

FROM dockerfile/java

RUN wget http://repo1.maven.org/maven2/org/eclipse/jetty/jetty-runner/9.2.0.M0/jetty-runner-9.2.0.M0.jar -O /root/jetty.jar

ADD target/dwc-services-0.0.4-standalone.war /root/dwc-services.war

EXPOSE 8080

CMD ["java -jar","/root/jetty.jar","/root/dwc-services.war"]

Dockerfile

FROM ubuntu:14.04

RUN apt-get update && \
    apt-get upgrade -y && \
    apt-get install apache2 libapache2-mod-php5 php5 php5-cli php5-curl php5-common php5-sqlite php5-mysql php5-pgsql php5-gd -y 

RUN apt-get install supervisor -y
RUN mkdir /var/log/supervisord 

RUN a2enmod rewrite 

RUN sed -i -e 's/memory_limit.*/memory_limit=512M/g' /etc/php5/apache2/php.ini && \
    sed -i -e 's/upload_max_filesize.*/upload_max_filesize=128M/g' /etc/php5/apache2/php.ini && \
    sed -i -e 's/post_max_size.*/post_max_size=128M/g' /etc/php5/apache2/php.ini && \
    sed -i -e 's/display_errors.*/display_erros=On/g' /etc/php5/apache2/php.ini

ENV APACHE_RUN_USER www-data
ENV APACHE_RUN_GROUP www-data
ENV APACHE_LOG_DIR /var/log/apache2
ENV APACHE_LOCK_DIR /var/lock/apache2
ENV APACHE_PID_FILE /var/apache2.pid
ENV PHP_ENV production

ADD default.conf /etc/apache2/sites-available/000-default.conf

ADD supervisord.conf /etc/supervisor/conf.d/apache.conf

ADD vendor /var/www/vendor
RUN chown www-data.www-data /var/www/vendor -Rf

ADD . /var/www
RUN chown www-data.www-data /var/www -Rf

EXPOSE 80
EXPOSE 9001

CMD ["supervisord"]

Problem

All that puts more pressure on Ops.

Dev Ops

Beyond docker.

docker-compose

etcd:
    image: coreos/etcd
    name: etcd
    ports:
       - "4001:4001"
logstash:
    image: diogok/logstash
    name: logstash
    ports:
        - 80
        - 9514
proxy:
    image: diogok/proxy-docker
    name: proxy
    environment:
        ROOT_APP: "connect"
    ports:
        - "80:80"
        - "9001"
    links:
        - "etcd:etcd"
connect:
    image: cncflora/connect
    name: connect
    ports:
        - "8080"
        - "9001"
    volumes:
        - "/var/floraconnect:/var/floraconnect:rw"
    environment:
        PROXY: "/connect"
dwc_services:
    image: cncflora/dwc-services
    name: dwc_services
    ports:
        - "8080"
        - "9001"
    environment:
        PROXY: "/dwc_services"
floradata:
    image: cncflora/floradata
    name: floradata
    ports:
        - "80"

Orchestrate your container.

ETCD

Share configuration REST server.

Nginx

Proxy to the container network

http {
    server {
        listen 80;

        # Proxying the connections connections, data come from ETCD
        <% for @app in @data.values %>
        location /<%= @app['name'] %>/ {
            rewrite /<%= @app['name'] %>/(.*) /$1 break;
            proxy_pass         http://<%= @app['networksettings']['ipaddress'] %>:<%= @app['networksettings']['ports'].keys.first.gsub("/tcp","") %>;
            proxy_redirect     default;
            proxy_set_header   Host $host;
            proxy_set_header   X-Real-IP $remote_addr;
            proxy_set_header   X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header   X-Forwarded-Host $server_name;
        }
        <% end %>
    }
}

Supervisord

SUpervvisor the processes

Dev & Ops

My personal state of art

Context:

Solving problems.

Problem:

Architectural style

Microservices

Brings to the architecture:

Polyglot programming

Polyglot persistence

Best tool for each job

Pressure on the Ops:

Automation

Monitoring

Provisioning

Fast delivery

Problem:

Archtectural style

REST

REST

Resumindo

In practice: HTTP

Problem:

GIT

Solves

Problem:

Many!

PHP

The language of the web

Pros

Cons

Best now a days:

Ruby

The new language of the Web!

Pros

Cons

How I use it

Clojure

Functional programming on the JVM

Prós

Cons

How I use it

Problem:

CouchDB

It is

ElasticSearch

It is

PostgreSQL

It is

SQLite

It is

Problem:

TDD

TDD

BDD

Problem:

Vagrant!

Vagrantfile

Vagrantfile

Problem:

Docker

It is

How?

Dockerfile

Dockerfile

Problem

Dev Ops

docker-compose

ETCD

Nginx

Supervisord

Logstash

Next step:

Continuous integration

Continuous delivery

THE END

http://diogok.net