Scott and John talk about how they implemented Docker @ RelateIQ

1. 10/17/2013 - Docker @ RelateIQ - Scott Bessler & John Fiedler

2. Docker @ RelateIQ
@
Scott Bessler
scott@relateiq.com
@scottbessler
John Fiedler
john@relateiq.com
@johnfiedler
Blog
blog.relateiq.com
RelateIQ Twitter
@relateiq
File Repo
https://github.com/relateiq/docker_public

3. Agenda
● Part 1 - What we did with Docker
● Part 2 - Why we want to replace Chef
● Part 3 - Future plans using Docker

4. Part 1. Things were getting out of control... Bring on hack day!
+
+
Hack Day!
=
=

5. What we needed to build
Docker Files
Minimum
Kafka
MongoDB
Redis
Cassandra
Zookeeper
Elasticsearch
Crushing IT
Docker Files
Storm
Jetty
Nginx
+ Private Docker Registry

6. End product
MAC OS X
$ devenv start
One
Command!
$ devenv-inner start
Kafka
VM
Containers
MongoDB
Redis
Cassandra
Zookeeper
ubuntu
Elasticsearch

7. Orchestration scripts
devenv.sh (outer)
[up|update|ssh]
●
●
Controls Vagrant
Controls inner script
devenv-inner.sh (inner)
[stop|start|kill|update|status|restart]
●
Controls Docker containers
update(){
start(){
apt-get update
mkdir -p $APPS/zookeeper/data
apt-get install -y lxc-docker
mkdir -p $APPS/zookeeper/logs
ZOOKEEPER=$(docker run
docker pull server:4444/zookeeper
-d
docker pull server:4444/redis
-p 2181:2181
docker pull server:4444/cassandra
-v $APPS/zookeeper/logs:/logs
docker pull server:4444/elasticsearch
server:4444/zookeeper)
docker pull server:4444/mongo
echo "Started ZOOKEEPER in
container $ZOOKEEPER"
echo "Wiring containers together… "
docker pull server:4444/kafka
}
kill/stop(){
(later slide)
echo "Killing all docker containers:"
docker ps | tail -n +2 |cut -d ' ' -f 1 | xargs docker kill
}

8. Dockerfile’s
Dockerfile
FROM server:4444/oracle-java7
Std Image
RUN apt-get update
RUN apt-get install -y git curl build-essential make gcc wget
RUN cd /opt && wget http://apache.mirrors.pair.com/cassandra/1.2.9/apachecassandra-1.2.9-bin.tar.gz
RUN cd /opt && tar zxf apache-cassandra-*.tar.gz
RUN rm /opt/*.tar.gz
RUN mv /opt/apache-cassandra-* /opt/cassandra
RUN apt-get install -y lsof
#!/bin/bash
echo "Cassandra node
configuration:"
echo $CASS_SEEDS
echo $CASS_TOKEN
start.sh
echo $CASS_LOCAL_IP
HOST=`hostname`
echo "127.0.0.1 $HOST" >>
/etc/hosts
/opt/cassandra/bin/cassandra -f
ADD cassandra.yaml /opt/cassandra/conf/cassandra.yaml
ADD cassandra-env.sh /opt/cassandra/conf/cassandra-env.sh
ADD log4j-server.properties /opt/cassandra/conf/log4j-server.properties
ADD start.sh /opt/cassandra/bin/start.sh
ADD cassandra-topology.properties /opt/cassandra/conf/cassandra-topology.
properties
RUN chmod 755 /opt/cassandra/bin/start.sh
RUN mkdir /logs
...
VOLUME [ "/logs" ]
...
EXPOSE 7000
….
CMD "/opt/cassandra/bin/start.sh"

9. Networking - port forwarding
Question:
Dude, where’s mongo?
Answer:
Port forwarding
Example:
> Docker file
expose 9999 (dockerfile)
> Docker
Machine
localhost:27019
vagrant
config.vm.forward_port 9999,27019
docker
run -p 9999:27018
docker run -p 8000:9999
> VAGRANT
config.vm.forward_port 27018, 8000
> localhost:8000
docker file
expose 27018
MongoDB
Tip:
Always use the same port if you can
Containers

10. Networking - connecting two containers
Question:
Kafka: Dude, where’s Zookeeper?
Answer:
Pipework
Machine
vagrant
https://github.com/jpetazzo/pipework
Example:
docker
> startup the container
KAFKA = docker run -e zooip=192.168.1.1
pipework br1
> Pipework
pipework br1 $KAFKA 192.168.1.2
pipework br1 $ZOOKEEPER 192.168.1.1
(dockerfile)
Zookeeper
Tip:
Use this for clustering
Kafka
192.168.1.1
192.168.1.2
Containers

11. Best practices
What we found
Machine
*Abstraction
●
●
●
●
Data
Logs
Don’t end with tailing
End with foreground
execution
● Start script for runtime
configuration
● 42 layers, combine
Dockerfile lines
● Up Vagrant RAM default
VM
/logs/mongo
/data/mongo
/logs/kafka
/data/kafka
*Standard Data and log dirs
*Static IP
MongoDB
Kafka
*Start Script
*Foreground Execution
*42 layers only
Containers

12. Chef
4 reasons why we want to replace Chef
with Docker in prod

13. 1. Dynamic Configuration Is Too Complex
Chef
● Dynamic configuration
through complex
attribute system
Docker Dockerfile
● Admittedly less powerful
● What you see is what you
get
● Environment variables
● Inspect environment at
any step

14. 2. External Dependencies Cause Flaky Provisioning
Chef
Docker
● Once it’s in, it’s in.
● Dependencies are
● Even if you change an
external forever
image, only incremental
● Slower. You pay the price
changes need to be sent
on every node deployed
to hosts
● Under load? Want a new
node? Uh oh, the file
download is missing.
● Node creation needs to be
foolproof
● Or just use chef-solo to
bake images

15. 3. Configuration Changes Create an Inconsistent State
Chef
Docker
● Configuration change? Try ● Ship configuration
changes as image deltas
to apply it to running node
● Nearly instant restarts
● Which changes restart
● Easier to be disciplined
which services?
and have each node be
● Removed something?
identical
○ knife ssh
○ tombstone
● Another reason to just bake
images

16. 4. Developers! Developers! Developers!
Chef
Dockerfile
● Typo? Fix it and start from
● Error? Throw it all out.
the previous successful
● Every iteration/test takes:
command
○ Boot time
● Containers encourage
○ Every step
single-purpose instances
● Can’t test every host
(put your monitoring on
● Can’t test every
the host)
combination of cookbooks
○ Monitoring
○ Logging

17. Part 3. Whats next? What do we want?
What’s Next for Docker @ RelateIQ?
● Replace Chef search (simple node database in
elasticsearch)
● Monitoring via StatsD and/or Datadog
Wish List
● Dockerfile repo/trusted builds (mentioned on dockerdev)
● Centralized Docker host management
● Mac host (no more virtualbox!)

18. We’re hiring! https://www.relateiq.com/jobs.html