Docker: Create Docker Swarm cluster with Consul Discovery

By | 19/04/2016

Introduction

In this post, we created a Swarm cluster with the internal Docker discovery service. As I read a lot about Consul, I also wanted to see how this interacts with Docker Swarm.

Creating consul host

WAUTERW-M-G007:~ wauterw$ docker-machine create -d virtualbox consul-host
Running pre-create checks...
Creating machine...
(consul-host) Copying /Users/wauterw/.docker/machine/cache/boot2docker.iso to /Users/wauterw/.docker/machine/machines/consul-host/boot2docker.iso...
(consul-host) Creating VirtualBox VM...
(consul-host) Creating SSH key...
(consul-host) Starting the VM...
(consul-host) Check network to re-create if needed...
(consul-host) Waiting for an IP...
Waiting for machine to be running, this may take a few minutes...
Detecting operating system of created instance...
Waiting for SSH to be available...
Detecting the provisioner...
Provisioning with boot2docker...
Copying certs to the local machine directory...
Copying certs to the remote machine...
Setting Docker configuration on the remote daemon...
Checking connection to Docker...
Docker is up and running!
To see how to connect your Docker Client to the Docker Engine running on this virtual machine, run: docker-machine env consul-host

We can then point our terminal to the newly created consul-host and install Consul itself.

WAUTERW-M-G007:bin wauterw$ eval $(docker-machine env consul-host)

Create the following ‘docker-compose.yml’ file:

myconsul:
  image: progrium/consul
  restart: always
  hostname: consul
  ports:
    - 8500:8500
  command: "-server -bootstrap"

And then run the ‘docker-compose up -d’ command:

Pulling myconsul (progrium/consul:latest)...
latest: Pulling from progrium/consul
c862d82a67a2: Pull complete
0e7f3c08384e: Pull complete
0e221e32327a: Pull complete
09a952464e47: Pull complete
60a1b927414d: Pull complete
4c9f46b5ccce: Pull complete
417d86672aa4: Pull complete
b0d47ad24447: Pull complete
fd5300bd53f0: Pull complete
a3ed95caeb02: Pull complete
d023b445076e: Pull complete
ba8851f89e33: Pull complete
5d1cefca2a28: Pull complete
Digest: sha256:8cc8023462905929df9a79ff67ee435a36848ce7a10f18d6d0faba9306b97274
Status: Downloaded newer image for progrium/consul:latest
Creating wauterw_myconsul_1

Creating Swarm master

Now that we have a Consul host up and running, we can continue to create the Docker Swarm.

WAUTERW-M-G007:~ wauterw$ docker-machine create -d virtualbox --swarm --swarm-master --swarm-discovery="consul://$(docker-machine ip consul-host):8500" --engine-opt="cluster-store=consul://$(docker-machine ip consul-host):8500" --engine-opt="cluster-advertise=eth1:2376" swarm-master
Running pre-create checks...
Creating machine...
(swarm-master) Copying /Users/wauterw/.docker/machine/cache/boot2docker.iso to /Users/wauterw/.docker/machine/machines/swarm-master/boot2docker.iso...
(swarm-master) Creating VirtualBox VM...
(swarm-master) Creating SSH key...
(swarm-master) Starting the VM...
(swarm-master) Check network to re-create if needed...
(swarm-master) Waiting for an IP...
Waiting for machine to be running, this may take a few minutes...
Detecting operating system of created instance...
Waiting for SSH to be available...
Detecting the provisioner...
Provisioning with boot2docker...
Copying certs to the local machine directory...
Copying certs to the remote machine...
Setting Docker configuration on the remote daemon...
Configuring swarm...
Checking connection to Docker...
Docker is up and running!
To see how to connect your Docker Client to the Docker Engine running on this virtual machine, run: docker-machine env swarm-master

We can then verify the machines that are up and running already:

WAUTERW-M-G007:~ wauterw$ docker-machine ls
NAME           ACTIVE   DRIVER       STATE     URL                         SWARM                   DOCKER    ERRORS
consul-host    *        virtualbox   Running   tcp://192.168.99.101:2376                           v1.11.0
swarm-master   -        virtualbox   Running   tcp://192.168.99.102:2376   swarm-master (master)   v1.11.0

Adding nodes to Swarm cluster

Now that we have a Consul host as well as a Swarm master, we can continue to add nodes:

WAUTERW-M-G007:~ wauterw$ docker-machine create -d virtualbox --swarm --swarm-discovery="consul://$(docker-machine ip consul-host):8500" --engine-opt="cluster-store=consul://$(docker-machine ip consul-host):8500" --engine-opt="cluster-advertise=eth1:2376" swarm-node-01
Running pre-create checks...
Creating machine...
(swarm-node-01) Copying /Users/wauterw/.docker/machine/cache/boot2docker.iso to /Users/wauterw/.docker/machine/machines/swarm-node-01/boot2docker.iso...
(swarm-node-01) Creating VirtualBox VM...
(swarm-node-01) Creating SSH key...
(swarm-node-01) Starting the VM...
(swarm-node-01) Check network to re-create if needed...
(swarm-node-01) Waiting for an IP...
Waiting for machine to be running, this may take a few minutes...
Detecting operating system of created instance...
Waiting for SSH to be available...
Detecting the provisioner...
Provisioning with boot2docker...
Copying certs to the local machine directory...
Copying certs to the remote machine...
Setting Docker configuration on the remote daemon...
Configuring swarm...
Checking connection to Docker...
Docker is up and running!
To see how to connect your Docker Client to the Docker Engine running on this virtual machine, run: docker-machine env swarm-node-01

and a second node:

WAUTERW-M-G007:~ wauterw$ docker-machine create -d virtualbox --swarm --swarm-discovery="consul://$(docker-machine ip consul-host):8500" --engine-opt="cluster-store=consul://$(docker-machine ip consul-host):8500" --engine-opt="cluster-advertise=eth1:2376" swarm-node-02
Running pre-create checks...
Creating machine...
(swarm-node-02) Copying /Users/wauterw/.docker/machine/cache/boot2docker.iso to /Users/wauterw/.docker/machine/machines/swarm-node-02/boot2docker.iso...
(swarm-node-02) Creating VirtualBox VM...
(swarm-node-02) Creating SSH key...
(swarm-node-02) Starting the VM...
(swarm-node-02) Check network to re-create if needed...
(swarm-node-02) Waiting for an IP...
Waiting for machine to be running, this may take a few minutes...
Detecting operating system of created instance...
Waiting for SSH to be available...
Detecting the provisioner...
Provisioning with boot2docker...
Copying certs to the local machine directory...
Copying certs to the remote machine...
Setting Docker configuration on the remote daemon...
Configuring swarm...
Checking connection to Docker...
Docker is up and running!
To see how to connect your Docker Client to the Docker Engine running on this virtual machine, run: docker-machine env swarm-node-02

If we then check all hosts, we see the complete list:

WAUTERW-M-G007:~ wauterw$ docker-machine ls
NAME            ACTIVE   DRIVER       STATE     URL                         SWARM                   DOCKER    ERRORS
consul-host     *        virtualbox   Running   tcp://192.168.99.101:2376                           v1.11.0
swarm-master    -        virtualbox   Running   tcp://192.168.99.102:2376   swarm-master (master)   v1.11.0
swarm-node-01   -        virtualbox   Running   tcp://192.168.99.103:2376   swarm-master            v1.11.0
swarm-node-02   -        virtualbox   Running   tcp://192.168.99.104:2376   swarm-master            v1.11.0

We can also look at some more info on the Docker Swarm cluster:

WAUTERW-M-G007:~ wauterw$ eval "$(docker-machine env --swarm swarm-master)"
WAUTERW-M-G007:~ wauterw$ docker info
Containers: 4
 Running: 4
 Paused: 0
 Stopped: 0
Images: 3
Server Version: swarm/1.2.0
Role: primary
Strategy: spread
Filters: health, port, dependency, affinity, constraint
Nodes: 3
 swarm-master: 192.168.99.102:2376
  └ Status: Healthy
  └ Containers: 2
  └ Reserved CPUs: 0 / 1
  └ Reserved Memory: 0 B / 1.021 GiB
  └ Labels: executiondriver=, kernelversion=4.1.19-boot2docker, operatingsystem=Boot2Docker 1.11.0 (TCL 7.0); HEAD : 32ee7e9 - Wed Apr 13 20:06:49 UTC 2016, provider=virtualbox, storagedriver=aufs
  └ Error: (none)
  └ UpdatedAt: 2016-04-21T08:06:19Z
  └ ServerVersion: 1.11.0
 swarm-node-01: 192.168.99.103:2376
  └ Status: Healthy
  └ Containers: 1
  └ Reserved CPUs: 0 / 1
  └ Reserved Memory: 0 B / 1.021 GiB
  └ Labels: executiondriver=, kernelversion=4.1.19-boot2docker, operatingsystem=Boot2Docker 1.11.0 (TCL 7.0); HEAD : 32ee7e9 - Wed Apr 13 20:06:49 UTC 2016, provider=virtualbox, storagedriver=aufs
  └ Error: (none)
  └ UpdatedAt: 2016-04-21T08:06:08Z
  └ ServerVersion: 1.11.0
 swarm-node-02: 192.168.99.104:2376
  └ Status: Healthy
  └ Containers: 1
  └ Reserved CPUs: 0 / 1
  └ Reserved Memory: 0 B / 1.021 GiB
  └ Labels: executiondriver=, kernelversion=4.1.19-boot2docker, operatingsystem=Boot2Docker 1.11.0 (TCL 7.0); HEAD : 32ee7e9 - Wed Apr 13 20:06:49 UTC 2016, provider=virtualbox, storagedriver=aufs
  └ Error: (none)
  └ UpdatedAt: 2016-04-21T08:05:56Z
  └ ServerVersion: 1.11.0
Plugins:
 Volume:
 Network:
Kernel Version: 4.1.19-boot2docker
Operating System: linux
Architecture: amd64
CPUs: 3
Total Memory: 3.064 GiB
Name: 6b2c27a806de
Docker Root Dir:
Debug mode (client): false
Debug mode (server): false
WARNING: No kernel memory limit support

Note: Take note of the fact that we used the –swarm flag in the previous command eval “$(docker-machine env –swarm swarm-master)”. The above command will address the entire cluster. You can also run eval “$(docker-machine env swarm-master)” but this will address the swarm-manager ‘host’, not the cluster.

WAUTERW-M-G007:~ wauterw$ eval "$(docker-machine env swarm-master)"
WAUTERW-M-G007:~ wauterw$ docker info
Containers: 2
 Running: 2
 Paused: 0
 Stopped: 0
Images: 1
Server Version: 1.11.0
Storage Driver: aufs
 Root Dir: /mnt/sda1/var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 12
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: null host bridge
Kernel Version: 4.1.19-boot2docker
Operating System: Boot2Docker 1.11.0 (TCL 7.0); HEAD : 32ee7e9 - Wed Apr 13 20:06:49 UTC 2016
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 996.1 MiB
Name: swarm-master
ID: X54E:OPXM:PMNZ:72KN:JBPU:6T6G:KA5L:YDFH:MYSB:KL5B:T4IE:7J5W
Docker Root Dir: /mnt/sda1/var/lib/docker
Debug mode (client): false
Debug mode (server): true
 File Descriptors: 26
 Goroutines: 71
 System Time: 2016-04-21T08:07:25.222630177Z
 EventsListeners: 1
Registry: https://index.docker.io/v1/
Labels:
 provider=virtualbox
Cluster store: consul://192.168.99.101:8500
Cluster advertise: 192.168.99.102:2376

Let’s also take a look at the individual nodes:

WAUTERW-M-G007:~ wauterw$ eval $(docker-machine env swarm-node-01)
WAUTERW-M-G007:~ wauterw$ docker info
Containers: 1
 Running: 1
 Paused: 0
 Stopped: 0
Images: 1
Server Version: 1.11.0
Storage Driver: aufs
 Root Dir: /mnt/sda1/var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 10
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: null host bridge
Kernel Version: 4.1.19-boot2docker
Operating System: Boot2Docker 1.11.0 (TCL 7.0); HEAD : 32ee7e9 - Wed Apr 13 20:06:49 UTC 2016
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 996.1 MiB
Name: swarm-node-01
ID: KKID:COTH:PXOE:RVKX:YC6R:QX2T:6OI7:3L55:MEWC:GBCO:4XEU:ARDI
Docker Root Dir: /mnt/sda1/var/lib/docker
Debug mode (client): false
Debug mode (server): true
 File Descriptors: 23
 Goroutines: 65
 System Time: 2016-04-21T08:13:40.624893775Z
 EventsListeners: 1
Registry: https://index.docker.io/v1/
Labels:
 provider=virtualbox
Cluster store: consul://192.168.99.101:8500
Cluster advertise: 192.168.99.103:2376

And then the second hosts:

WAUTERW-M-G007:~ wauterw$ eval $(docker-machine env swarm-node-02)
WAUTERW-M-G007:~ wauterw$ docker info
Containers: 1
 Running: 1
 Paused: 0
 Stopped: 0
Images: 1
Server Version: 1.11.0
Storage Driver: aufs
 Root Dir: /mnt/sda1/var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 10
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge null host
Kernel Version: 4.1.19-boot2docker
Operating System: Boot2Docker 1.11.0 (TCL 7.0); HEAD : 32ee7e9 - Wed Apr 13 20:06:49 UTC 2016
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 996.1 MiB
Name: swarm-node-02
ID: VEQO:6VTZ:ZEPV:L3XB:62QF:V7OJ:DYAX:OO6S:ZWOA:2ULI:4I4H:2TIJ
Docker Root Dir: /mnt/sda1/var/lib/docker
Debug mode (client): false
Debug mode (server): true
 File Descriptors: 23
 Goroutines: 65
 System Time: 2016-04-21T08:14:04.588596253Z
 EventsListeners: 1
Registry: https://index.docker.io/v1/
Labels:
 provider=virtualbox
Cluster store: consul://192.168.99.101:8500
Cluster advertise: 192.168.99.104:2376

So, we see that across all three hosts (Swarm-manager, Swarm-node-01 and Swarm-node-02) that we have 4 containers in total. To see what they are, do the following:

WAUTERW-M-G007:~ wauterw$ eval "$(docker-machine env --swarm swarm-master)"
WAUTERW-M-G007:~ wauterw$ docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
WAUTERW-M-G007:~ wauterw$ docker ps -a
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS                     PORTS                                     NAMES
85ad1823c909        swarm               "/swarm list consul:/"   4 minutes ago       Exited (0) 4 minutes ago                                             swarm-master/sharp_saha
65a93f731272        swarm:latest        "/swarm join --advert"   12 minutes ago      Up 12 minutes              2375/tcp                                  swarm-node-02/swarm-agent
ec757c42ab59        swarm:latest        "/swarm join --advert"   13 minutes ago      Up 13 minutes              2375/tcp                                  swarm-node-01/swarm-agent
b8c7a7baf2dc        swarm:latest        "/swarm join --advert"   19 minutes ago      Up 19 minutes              2375/tcp                                  swarm-master/swarm-agent
6b2c27a806de        swarm:latest        "/swarm manage --tlsv"   19 minutes ago      Up 19 minutes              2375/tcp, 192.168.99.102:3376->3376/tcp   swarm-master/swarm-agent-master

Last item for this topic, we can also query the consul host to list all nodes that are part of the cluster:

WAUTERW-M-G007:~ wauterw$ docker run swarm list consul://$(docker-machine ip consul-host):8500
time="2016-04-21T08:11:48Z" level=info msg="Initializing discovery without TLS"
192.168.99.102:2376
192.168.99.103:2376
192.168.99.104:2376

And last but not least, you can also take a look at the consul UI. In our example, this is running on http://192.168.99.101:8500/ui. Below a screenshot of what you can expect.
Consul1

This post described how to get a Swarm cluster up and running with Consul Service Discovery. In a next post, we will install some application on this cluster.

Leave a Reply

Your email address will not be published. Required fields are marked *