Docker recently started pushing towards a complete solution for orchestrating distributed applications. This post shows you how to set up a Swarm cluster on the Google Cloud Platform using Docker Machine and then run a Crate cluster using Docker Swarm. First and foremost, you'll have to create the firewall rules so Swarm is able to talk to its instances. Unfortunately it's using the public IP addresses for communication so we have to allow certain ports to be used. You can start up the Swarm cluster from your personal laptop, but I prefer doing it from the cloud, so everyone in the team may join the in the fun. The script `startup-master.sh` will install Docker, docker-machine, and all necessary dependencies when launching the instance. Wait until the instance is up and running (go grab a coffee) and then connect to it using the `gcloud compute ssh` command. To create a Swarm token - a unique token used to identify the Swarm cluster - We call the environment for the token `env-crate`. As the output already suggests, we load the environment into the shell, and then run `swarm create` which will generate a Swarm. I found it useful to put the token into bash profile, so it's available as soon as you log on to the machine. The Swarm master is the first instance of the cluster, and all other subsequent nodes will connect to that. We use the `google` driver and `n1-standard-2` instances for a bit more computing power, also a bigger boot disk size than default. The option `--swarm-master`` is the key here: Create Swarm nodes basically works the same way, just take the command from above, omit the `--swarm-master` option and name your instance appropriately. ... and repeat for `crate-swarm-node2` and `crate-swarm-node3` ... Now we have a Swarm cluster of 4 nodes (1 master, 3 nodes) that we can connect to: Ok, the Swarm cluster is ready to be used. Now let's start a Crate instance in this cluster. You can simply do that with the regular The Crate instance was automatically deployed on one of the available nodes. To verify whether the Crate node has started correctly you can look at the logs or execute a simple curl command. So far we've started a single Crate instance that does not need to communicate with any other nodes. Google Cloud Platform (and other cloud hosting network environments) do not have multicast enabled, so Crate needs to use unicast discovery, Usually you know on which host you're running your Crate container, so you can pass the hostname in the We're really looking forward to seeing a solution for this problem. In the meantime, I've come up with an idea how to easily deploy Crate on all Swarm nodes: This is how the bash script looks: Having that in place, you can simply run this bash script and a Crate container will be started on each Swarm node. It might take some time until the nodes will find each other. Here we go. The Crate cluster is up and running. You can also visit the web-based administration UI which is available on every node on port 4200 Setting up a cluster to be able to deploy your Crate database cluster might sound difficult, Once the Swarm cluster is up and running it is easy to add and remove instances I am really looking forward to seeing features that allows you to you know on which machine a container is deployed so this information can be fed into the container. Unfortunately this is required to make the Crate deployment using Swarm simpler.
Machine and Swarm are two projects that let you manage cluster instances and deploy Docker containers onto them.
All instructions in this post refer to docker-machine version 0.1.0-rc5.Prerequisites
# ssh from everywhere
$ gcloud compute firewall-rules create default-ssh --allow tcp:22 icmp --source-range 0.0.0.0/0
$ gcloud compute firewall-rules create default-swarm --allow tcp:2376 tcp:3376 --source-range 0.0.0.0/0
$ gcloud compute firewall-rules create default-crate --allow tcp:4200 tcp:4300 --source-range 0.0.0.0/0
This means we have to create a `machine-master` first and that instance is also the place where later on, all the deployment commands are executed.# ssh from everywhere
$ gcloud compute firewall-rules create default-ssh --allow tcp:22 icmp --source-range 0.0.0.0/0
$ gcloud compute firewall-rules create default-swarm --allow tcp:2376 tcp:3376 --source-range 0.0.0.0/0
$ gcloud compute firewall-rules create default-crate --allow tcp:4200 tcp:4300 --source-range 0.0.0.0/0
#!/bin/bash -x
sudo su -
yum clean expire-cache
yum update -y
yum install -y wget docker bridge-utils
curl -Lo /usr/bin/docker-machine https://github.com/docker/machine/releases/download/v0.1.0-rc5/docker-machine_linux-amd64
chmod +x /usr/bin/docker-machine
Creating The Swarm Cluster
We'll need the Google Project ID several times, so it's a good idea to set the environment variable too.$ gcloud compute ssh machine-master --zone us-central1-a
$ echo "export GCE_PROJECT=<YOUR_PROJECT_ID>" >> .bash_profile
$ source .bash_profile
Obtaining The Cluster Token
you need to create a remote Docker host first, and then obtain the Swarm token on that Docker host.$ docker-machine create -d google --google-project ${GCE_PROJECT} env-crate
INFO[0000] Opening auth URL in browser.
INFO[0000] https://accounts.google.com/o/oauth2/auth?client_id=...
INFO[0000] If the URL doesn't open please open it manually and copy the code here.
INFO[0008] Got code: ....
INFO[0008] Saving token in /home/christian/.docker/machines/env-crate/gce_token
INFO[0008] Creating host...
INFO[0008] Generating SSH Key
INFO[0008] Creating instance.
INFO[0010] Waiting for Instance...
INFO[0027] Waiting for SSH...
INFO[0054] Uploading SSH Key
INFO[0054] Waiting for SSH Key
INFO[0058] Configuring Machine...
INFO[0899] "env-crate" has been created and is now the active machine.
INFO[0899] To point your Docker client at it, run this in your shell: $(docker-machine env env-crate)
The hash of the container is used as the token.$ $(docker-machine env env-crate)
$ docker run swarm create
572c97e72aefe953031f5f2e7d3ca9bb
$ echo "export TOKEN=572c97e72aefe953031f5f2e7d3ca9bb" >> .bash_profile
$ source .bash_profile
Creating Master and Nodes
However, the master also acts as a regular node in the cluster, meaning that Docker containers can be launched on it.$ docker-machine create -d google --google-project=${GCE_PROJECT} --google-machine-type=n1-standard-2 --google-disk-size=50 --swarm --swarm-master --swarm-discovery token://${TOKEN} crate-swarm
INFO[0000] Opening auth URL in browser.
INFO[0000] https://accounts.google.com/o/oauth2/auth?client_id=...
INFO[0000] If the URL doesn't open please open it manually and copy the code here.
INFO[0035] Got code: ...
INFO[0035] Saving token in /home/christian/.docker/machines/swarm-master/gce_token
INFO[0035] Creating host...
INFO[0036] Generating SSH Key
INFO[0036] Creating instance.
INFO[0037] Waiting for Instance...
INFO[0055] Waiting for SSH...
INFO[0119] Uploading SSH Key
INFO[0119] Waiting for SSH Key
INFO[0122] Configuring Machine...
INFO[0565] Configuring Swarm...
INFO[0574] "swarm-master" has been created and is now the active machine.
INFO[0574] To point your Docker client at it, run this in your shell: $(docker-machine env swarm-master)$ docker-machine create -d google --google-project=${GCE_PROJECT} --google-machine-type=n1-standard-2 --google-disk-size=50 --swarm --swarm-discovery token://${TOKEN} crate-swarm-node1
INFO[0000] Opening auth URL in browser.
INFO[0000] https://accounts.google.com/o/oauth2/auth?client_id=...
INFO[0000] If the URL doesn't open please open it manually and copy the code here.
INFO[0015] Got code: ...
INFO[0015] Saving token in /home/christian/.docker/machines/swarm-node-1/gce_token
INFO[0015] Creating host...
INFO[0015] Generating SSH Key
INFO[0015] Creating instance.
INFO[0017] Waiting for Instance...
INFO[0036] Waiting for SSH...
INFO[0054] Uploading SSH Key
INFO[0055] Waiting for SSH Key
INFO[0057] Configuring Machine...
INFO[0498] Configuring Swarm...
INFO[0510] "crate-swarm-node1" has been created and is now the active machine.
INFO[0510] To point your Docker client at it, run this in your shell: $(docker-machine env crate-swarm-node1)# inspect the nodes
$ docker-machine ls
NAME ACTIVE DRIVER STATE URL SWARM
crate-swarm google Running tcp://104.154.93.89:2376 crate-swarm (master)
crate-swarm-node1 google Running tcp://104.154.73.244:2376 crate-swarm
crate-swarm-node2 google Running tcp://107.178.213.253:2376 crate-swarm
crate-swarm-node3 * google Running tcp://104.197.14.121:2376 crate-swarm
env-crate google Running tcp://104.154.85.254:2376
# load crate-swarm environment into shell
$ $(docker-machine env --swarm crate-swarm)
$ docker info
Containers: 5
Images: 0
Storage Driver:
Nodes: 4
crate-swarm: 104.154.93.89:2376
└ Containers: 2
└ Reserved CPUs: 0 / 2
└ Reserved Memory: 0 B / 7.306 GiB
crate-swarm-node1: 104.154.73.244:2376
└ Containers: 1
└ Reserved CPUs: 0 / 2
└ Reserved Memory: 0 B / 7.306 GiB
crate-swarm-node2: 107.178.213.253:2376
└ Containers: 1
└ Reserved CPUs: 0 / 2
└ Reserved Memory: 0 B / 7.306 GiB
crate-swarm-node3: 104.197.14.121:2376
└ Containers: 1
└ Reserved CPUs: 0 / 2
└ Reserved Memory: 0 B / 7.306 GiB
Execution Driver:
Kernel Version:
Operating System:
Launching Crate
docker run command that we all know and love.$ docker run -d -p 4200:4200 --env CRATE_HEAP_SIZE=1g crate:latest crate -Des.cluster.name=hello-swarm
3cc561fc48a4f4bc983410b15bc1e9ffd13143c4bc482a2b2e064d3dfbfc297f
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
3cc561fc48a4 crate:latest "crate -Des.cluster. 17 seconds ago Up 6 seconds 4300/tcp, 104.197.14.121:4200->4200/tcp crate-swarm-node3/hopeful_babbage
You can find out which one by looking at the port mapping, in this case on 104.197.14.121 which is the host crate-swarm-node3.$ docker logs 3cc561fc48a4
[2015-02-26 15:36:07,518][INFO ][node ] [Theresa Cassidy] version[1.4.2], pid[1], build[${build/NA]
[2015-02-26 15:36:07,519][INFO ][node ] [Theresa Cassidy] initializing ...
[2015-02-26 15:36:07,579][INFO ][plugins ] [Theresa Cassidy] loaded [crate-core], sites [crate-admin]
[2015-02-26 15:36:07,899][INFO ][io.crate.module.CrateCoreModule] configuring crate. version: 0.47.4
[2015-02-26 15:36:10,811][INFO ][io.crate.rest.CrateRestFilter] Elasticsearch HTTP REST API not enabled
[2015-02-26 15:36:10,915][INFO ][node ] [Theresa Cassidy] initialized
[2015-02-26 15:36:10,916][INFO ][node ] [Theresa Cassidy] starting ...
[2015-02-26 15:36:10,917][INFO ][io.crate.blob.BlobService] [Theresa Cassidy] BlobService.doStart() io.crate.blob.BlobService@50a4e1e6
[2015-02-26 15:36:10,980][INFO ][http ] [Theresa Cassidy] bound_address {inet[/0:0:0:0:0:0:0:0:4200]}, publish_address {inet[/172.17.0.18:4200]}
[2015-02-26 15:36:10,996][INFO ][transport ] [Theresa Cassidy] bound_address {inet[/0:0:0:0:0:0:0:0:4300]}, publish_address {inet[/172.17.0.18:4300]}
[2015-02-26 15:36:11,010][INFO ][discovery ] [Theresa Cassidy] hello-swarm/CZ02-FrDSpi2QC2aGirwCg
[2015-02-26 15:36:14,780][INFO ][cluster.service ] [Theresa Cassidy] new_master [Theresa Cassidy][CZ02-FrDSpi2QC2aGirwCg][3cc561fc48a4][inet[/172.17.0.18:4300]]{http_address=http://172.17.0.18:4200}, reason: zen-disco-join (elected_as_master)
[2015-02-26 15:36:14,811][INFO ][node ] [Theresa Cassidy] started
[2015-02-26 15:36:14,907][INFO ][gateway ] [Theresa Cassidy] recovered [0] indices into cluster_state
$ curl -XPOST 104.197.14.121:4200/_sql?pretty -d '{"stmt":"select id, name, hostname from sys.nodes"}'
{
"cols" : [ "id", "name", "hostname" ],
"duration" : 4,
"rows" : [ [ "CZ02-FrDSpi2QC2aGirwCg", "Theresa Cassidy", "3cc561fc48a4" ] ],
"rowcount" : 1
}
Launching a Crate Cluster
Not we want to start multiple Crate nodes that form a cluster. Unfortunately, we're hitting a problem with Swarm here.
and because Crate is running inside a container, the Crate node would need to publish the IP or hostname of its host.docker run command.
However, when using Swarm you don't know where your container will be deployed when you execute the docker run command.
minimum_master_nodes setting from the amount of available Docker host (nodes/2+1)docker run explicitly on each host#!/bin/bash -x
export LIST=$(docker run -d swarm list token://$TOKEN)
export REMOTE_DOCKER_HOSTS=$(docker logs $LIST)
export MIN_MASTER_NODES=$(( $(docker logs $LIST | wc -l) / 2 + 1))
export UNICAST_HOSTS=$(docker logs $LIST | sed "s/:2376/:4300/" | paste -s -d",")
for host in $REMOTE_DOCKER_HOSTS; do
pub_host=$(echo $host | sed "s/:2376//")
docker -H tcp://$host \
run -d -p 4200:4200 -p 4300:4300 \
crate:latest \
crate -Des.cluster.name=crate-swarm \
-Des.multicast.enabled=false \
-Des.transport.publish_host=$pub_host \
-Des.discovery.zen.ping.unicast.hosts=$UNICAST_HOSTS \
-Des.discovery.zen.minimum_master_nodes=$MIN_MASTER_NODES
sleep 3
done
docker stop $LIST
docker rm $LIST
Finally, use curl to verify the cluster:$ curl -XPOST 104.197.14.121:4200/_sql?pretty -d '{"stmt":"select id, name, hostname from sys.nodes"}'
{
"cols" : [ "id", "name", "hostname" ],
"duration" : 4,
"rows" : [ ... ],
"rowcount" : 4
}
at the path /admin.Summary
but I have to say that Docker made a lot of effort to make this process as simple as possible.
(although it takes a long time sometimes until a new instance is provisioned and ready to use).Resources