Virtualized Docker Swarm


Why not? In reality, I always wanted to play with clustering, originally with proxmox and ceph, but I never had enough hardware to do so. I do however have a proxmox node with enough ram that I can host multiple lightweight nodes.

Docker swarm is lightweight enough that I can virtualize the entire cluster on my single proxmox host. While this isn’t fault tolerant like a cluster across multiple nodes, it does mean I can reboot cluster nodes for kernel updates and maintain my uptime. I also am able to add additional docker swarm nodes on separate hardware if I get additional hardware, and there is the benefit of having the cluster load balance itself for which software is running on which node.


Each node in the cluster is identical, each can be replaced by following the exact same process and while I don’t have automated deployment of new nodes, they are still closer to cattle than many of my other virtual machines. Due to the goal of replicated storage between the nodes, I should also be able to take a single node and rebuild the entire cluster if needed, since it would have the entire clusters configuration.


I’ll be demonstrating on a small cluster with 3 nodes, each of them a master. This does mean that we always need at least two running nodes for quorum, however this also means that we can reboot or replace any individual node and the cluster remains viable. Each of these will be running on Ubuntu Server 18.04. So for that, you’ll need:

  • Three running and updated instances of Ubuntu Server 18.04.
  • A user with sudo privileges.

That’s all you need to make this work.

Before doing anything, lets get these boxes up to date

sudo apt-get update
sudo apt-get dist-upgrade -y

If needed, reboot for kernel updates

Network Setup

Add some stuff about static IPs

We now need to map our IP addresses in /etc/hosts. Do this on each machine. Issue the command:

sudo nano /etc/hosts

In that file (on each machine), you’ll add something like this to the bottom of the file: docker-node01 docker-node02 docker-node03

Make sure to edit the above to match your IP addresses and hostnames.

Save and close the file.

Docker Swarm Setup

If you haven’t already done so, you need to install and deploy the Docker Swarm. On each machine install Docker with the command:

sudo apt-get install -y

Start and enable Docker with the commands:

sudo systemctl start docker
sudo systemctl enable docker

Add your user to the docker group (on all machines) with the command:

sudo usermod -aG docker $USER

Issue the following command (on all machines) so the changes take effect:

sudo newgrp docker

Next, we need to initialize the swarm. On the master issue the command:

docker swarm init --advertise-addr MASTER_IP

Where MASTER_IP is the IP address of the master.

Once the swarm has been initialized, you’ll need to get the manager join token using the following command, then execute the join command on all remaining nodes.

docker swarm join-token manager
docker swarm join --token SWMTKN-1-09c0p3304ookcnibhg3lp5ovkjnylmxwjac9j5puvsj2wjzhn1-2vw4t2474ww1mbq4xzqpg0cru

Copy that command and paste it into the terminal window of the nodes to join them to the master.

And that’s all there is to deploying the swarm.


You now need to install GlusterFS on each server within the swarm. First, install the necessary dependencies with the command:

sudo apt-get install software-properties-common -y

Next, add the necessary repository with the command:

sudo add-apt-repository ppa:gluster/glusterfs-3.12

Update apt with the command:

sudo apt-get update

Install the GlusterFS server with the command:

sudo apt install glusterfs-server -y

Finally, start and enable GlusterFS with the commands:

sudo systemctl start glusterd
sudo systemctl enable glusterd

Probing the Gluster Nodes

Now we’re going to have Gluster probe all of the nodes. This will be done from node 01. I’m going to stick with my example of with 3 nodes total in the cluster. Before you issue the command, you’ll need to change to the superuser with:

sudo -s

If you don’t issue the Gluster probe command from root, you’ll get an error that it cannot write to the logs. The probe command looks like:

gluster peer probe docker-node02; gluster peer probe docker-node03;

Make sure to edit the command to fit your configuration (for hostnames).

Once the command completes, you can check to make sure your nodes are connected with the command:

gluster pool list

You should see all nodes listed as connected

administrator@docker-01:/$ sudo gluster pool list
UUID                                    Hostname        State
48ea5f6a-fb12-4a34-a957-a051bc96ed67    bot-docker-02   Connected
ab4a7508-9e53-4118-84ef-d63f41dbba3a    bot-docker-03   Connected
a92c8669-441f-4a67-8a2f-b5d5a7e5cc27    localhost       Connected

Exit out of the root user with the exit command.

Create the Gluster Volume

Let’s create a directory to be used for the Gluster volume. This same command will be run on all machines:

sudo mkdir -p /gluster/volume1

Use whatever name you want in place of volume1.

Now we’ll create the volume across the cluster with the command (run only on the master):

sudo gluster volume create staging-gfs replica 3 docker-node01:/gluster/volume1 docker-node02:/gluster/volume1 docker-node03:/gluster/volume1 force

This creates a gluster cluster with each node having a full set of the data (3 replicas), using each of the nodes for the cluster. All the data in the gluster volume will be replicated between the nodes (in my case, over the 10g vswitch in proxmox).

Start the volume with the command:

sudo gluster volume start staging-gfs

The volume is now up and running, but we need to make sure the volume will mount on a reboot (or other circumstances). We’ll mount the volume to the /mnt directory. To do this, issue the following commands on all machines:

sudo -s
echo 'localhost:/staging-gfs /mnt glusterfs defaults,_netdev,backupvolfile-server=localhost 0 0' >> /etc/fstab
mount.glusterfs localhost:/staging-gfs /mnt
chown -R root:docker /mnt

To make sure the Gluster volume is mounted, issue the command:

df -h

You should see it listed at the bottom.

localhost:/staging-gfs   32G   11G   20G  37% /mnt

You can now create new files in the /mnt directory and they’ll show up in the /gluster/volume1 directories on every machine.

Using the new Gluster Volume with Docker

At this point, you are ready to integrate your persistent storage volume with docker. Say, for instance, you need persistent storage for a MySQL database. In your docker YAML files, you could add a section like so:

   - type: bind
     source: /mnt/staging_mysql
     target: /opt/mysql/data

Or in the case that you need an Inspircd server with persistent storage for the configuration:

    image: inspircd/inspircd-docker
      - /mnt/data/Inspircd:/inspircd/conf/

Since we’ve mounted our persistent storage in /mnt everything saved there on one docker node will sync with all other nodes.

And that’s how you can create persistent storage and then use it within a Docker Swarm cluster. Of course, this isn’t the only way to make persistent storage work, but it is one of the easiest (and cheapest). Give GlusterFS a try as your persistent storage option and see if it doesn’t work out for you.

Add a few services and see how it works. My nodes are mostly 4GB ram and 64GB storage which is running everything below.