{"id":281,"date":"2020-08-21T01:19:54","date_gmt":"2020-08-21T01:19:54","guid":{"rendered":"https:\/\/lab.rapternet.us\/?p=281"},"modified":"2020-08-23T22:11:29","modified_gmt":"2020-08-23T22:11:29","slug":"virtualized-docker-swarm","status":"publish","type":"post","link":"https:\/\/lab.rapternet.us\/?p=281","title":{"rendered":"Virtualized Docker Swarm"},"content":{"rendered":"\n<h2 class=\"wp-block-heading\">Why<\/h2>\n\n\n\n<p>Why not? In reality, I always wanted to play with clustering, originally with proxmox and ceph, but I never had enough hardware to do so. I do however have a proxmox node with enough ram that I can host multiple lightweight nodes.<\/p>\n\n\n\n<p>Docker swarm is lightweight enough that I can virtualize the entire cluster on my single proxmox host. While this isn&#8217;t fault tolerant like a cluster across multiple nodes, it does mean I can reboot cluster nodes for kernel updates and maintain my uptime. I also am able to add additional docker swarm nodes on separate hardware if I get additional hardware, and there is the benefit of having the cluster load balance itself for which software is running on which node.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Benefits<\/h2>\n\n\n\n<p>Each node in the cluster is identical, each can be replaced by following the exact same process and while I don&#8217;t have automated deployment of new nodes, they are still closer to cattle than many of my other virtual machines. Due to the goal of replicated storage between the nodes, I should also be able to take a single node and rebuild the entire cluster if needed, since it would have the entire clusters configuration.<\/p>\n\n\n\n<!--more-->\n\n\n\n<h2 class=\"wp-block-heading\">Needs<\/h2>\n\n\n\n<p>I\u2019ll be demonstrating on a small cluster with 3 nodes, each of them a master. This does mean that we always need at least two running nodes for quorum, however this also means that we can reboot or replace any individual node and the cluster remains viable. Each of these will be running on Ubuntu Server 18.04. So for that, you\u2019ll need:<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>Three running and updated instances of Ubuntu Server 18.04.<\/li><li>A user with sudo privileges.<\/li><\/ul>\n\n\n\n<p>That\u2019s all you need to make this work.<\/p>\n\n\n\n<p>Before doing anything, lets get these boxes up to date<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">sudo apt-get update<\/pre>\n\n\n\n<pre class=\"wp-block-preformatted\">sudo apt-get dist-upgrade -y<\/pre>\n\n\n\n<p>If needed, reboot for kernel updates<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Network Setup<\/h2>\n\n\n\n<p><strong>Add some stuff about static IPs<\/strong><\/p>\n\n\n\n<p>We now need to map our IP addresses in \/etc\/hosts. Do this on each machine. Issue the command:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">sudo nano \/etc\/hosts<\/pre>\n\n\n\n<p>In that file (on each machine), you\u2019ll add something like this to the bottom of the file:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">192.168.1.123 docker-node01\n192.168.1.124 docker-node02\n192.168.1.125 docker-node03<\/pre>\n\n\n\n<p>Make sure to edit the above to match your IP addresses and hostnames.<\/p>\n\n\n\n<p>Save and close the file.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Docker Swarm Setup<\/h2>\n\n\n\n<p>If you haven\u2019t already done so, you need to install and deploy the Docker Swarm. On each machine install Docker with the command:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">sudo apt-get install docker.io -y<\/pre>\n\n\n\n<p>Start and enable Docker with the commands:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">sudo systemctl start docker\nsudo systemctl enable docker<\/pre>\n\n\n\n<p>Add your user to the docker group (on all machines) with the command:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">sudo usermod -aG docker $USER<\/pre>\n\n\n\n<p>Issue the following command (on all machines) so the changes take effect:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">sudo newgrp docker<\/pre>\n\n\n\n<p>Next, we need to initialize the swarm. On the master issue the command:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">docker swarm init --advertise-addr MASTER_IP<\/pre>\n\n\n\n<p>Where MASTER_IP is the IP address of the master.<\/p>\n\n\n\n<p>Once the swarm has been initialized, you&#8217;ll need to get the manager join token using the following command, then execute the join command on all remaining nodes.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">docker swarm join-token manager\ndocker swarm join --token SWMTKN-1-09c0p3304ookcnibhg3lp5ovkjnylmxwjac9j5puvsj2wjzhn1-2vw4t2474ww1mbq4xzqpg0cru 192.168.1.123:2377<\/pre>\n\n\n\n<p>Copy that command and paste it into the terminal window of the nodes to join them to the master.<\/p>\n\n\n\n<p>And that\u2019s all there is to deploying the swarm.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">GlusterFS<\/h2>\n\n\n\n<p>You now need to install GlusterFS on each server within the swarm. First, install the necessary dependencies with the command:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">sudo apt-get install software-properties-common -y<\/pre>\n\n\n\n<p>Next, add the necessary repository with the command:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">sudo add-apt-repository ppa:gluster\/glusterfs-3.12<\/pre>\n\n\n\n<p>Update apt with the command:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">sudo apt-get update<\/pre>\n\n\n\n<p>Install the GlusterFS server with the command:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">sudo apt install glusterfs-server -y<\/pre>\n\n\n\n<p>Finally, start and enable GlusterFS with the commands:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">sudo systemctl start glusterd\nsudo systemctl enable glusterd<\/pre>\n\n\n\n<h2 class=\"wp-block-heading\">Probing the Gluster Nodes<\/h2>\n\n\n\n<p>Now we\u2019re going to have Gluster probe all of the nodes. This will be done from node 01. I\u2019m going to stick with my example of with 3 nodes total in the cluster. Before you issue the command, you\u2019ll need to change to the superuser with:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">sudo -s<\/pre>\n\n\n\n<p>If you don\u2019t issue the Gluster probe command from root, you\u2019ll get an error that it cannot write to the logs. The probe command looks like:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">gluster peer probe docker-node02; gluster peer probe docker-node03;<\/pre>\n\n\n\n<p>Make sure to edit the command to fit your configuration (for hostnames).<\/p>\n\n\n\n<p>Once the command completes, you can check to make sure your nodes are connected with the command:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">gluster pool list<\/pre>\n\n\n\n<p>You should see all nodes listed as connected<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">administrator@docker-01:\/$ sudo gluster pool list\nUUID                                    Hostname        State\n48ea5f6a-fb12-4a34-a957-a051bc96ed67    bot-docker-02   Connected\nab4a7508-9e53-4118-84ef-d63f41dbba3a    bot-docker-03   Connected\na92c8669-441f-4a67-8a2f-b5d5a7e5cc27    localhost       Connected<\/pre>\n\n\n\n<p>Exit out of the root user with the <strong>exit<\/strong> command.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Create the Gluster Volume<\/h2>\n\n\n\n<p>Let\u2019s create a directory to be used for the Gluster volume. This same command will be run on all machines:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">sudo mkdir -p \/gluster\/volume1<\/pre>\n\n\n\n<p>Use whatever name you want in place of volume1.<\/p>\n\n\n\n<p>Now we\u2019ll create the volume across the cluster with the command (run only on the master):<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">sudo gluster volume create staging-gfs replica 3 docker-node01:\/gluster\/volume1 docker-node02:\/gluster\/volume1 docker-node03:\/gluster\/volume1 force<\/pre>\n\n\n\n<p>This creates a gluster cluster with each node having a full set of the data (3 replicas), using each of the nodes for the cluster. All the data in the gluster volume will be replicated between the nodes (in my case, over the 10g vswitch in proxmox).<\/p>\n\n\n\n<p>Start the volume with the command:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">sudo gluster volume start staging-gfs<\/pre>\n\n\n\n<p>The volume is now up and running, but we need to make sure the volume will mount on a reboot (or other circumstances). We\u2019ll mount the volume to the \/mnt directory. To do this, issue the following commands on all machines:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">sudo -s\necho 'localhost:\/staging-gfs \/mnt glusterfs defaults,_netdev,backupvolfile-server=localhost 0 0' &gt;&gt; \/etc\/fstab\nmount.glusterfs localhost:\/staging-gfs \/mnt\nchown -R root:docker \/mnt\nexit<\/pre>\n\n\n\n<p>To make sure the Gluster volume is mounted, issue the command:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">df -h<\/pre>\n\n\n\n<p>You should see it listed at the bottom.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">localhost:\/staging-gfs   32G   11G   20G  37% \/mnt<\/pre>\n\n\n\n<p> You can now create new files in the <strong>\/mnt<\/strong> directory and they\u2019ll show up in the <strong>\/gluster\/volume1<\/strong> directories on every machine.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Using the new Gluster Volume with Docker<\/h2>\n\n\n\n<p>At this point, you are ready to integrate your persistent storage volume with docker. Say, for instance, you need persistent storage for a MySQL database. In your docker YAML files, you could add a section like so:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"> volumes:\n   - type: bind\n     source: \/mnt\/staging_mysql\n     target: \/opt\/mysql\/data<\/pre>\n\n\n\n<p>Or in the case that you need an Inspircd server with persistent storage for the configuration:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">services:\n  inspircd:\n    image: inspircd\/inspircd-docker\n    volumes:\n      - \/mnt\/data\/Inspircd:\/inspircd\/conf\/<\/pre>\n\n\n\n<p>Since we\u2019ve mounted our persistent storage in <strong>\/mnt<\/strong> everything saved there on one docker node will sync with all other nodes.<\/p>\n\n\n\n<p>And that\u2019s how you can create persistent storage and then use it within a Docker Swarm cluster. Of course, this isn\u2019t the only way to make persistent storage work, but it is one of the easiest (and cheapest). Give GlusterFS a try as your persistent storage option and see if it doesn\u2019t work out for you.<\/p>\n\n\n\n<p>Add a few services and see how it works. My nodes are mostly 4GB ram and 64GB storage which is running everything below.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"735\" height=\"1024\" src=\"https:\/\/lab.rapternet.us\/wp-content\/uploads\/2020\/08\/cluster-portainer-viz-735x1024.png\" alt=\"\" class=\"wp-image-286\" srcset=\"https:\/\/lab.rapternet.us\/wp-content\/uploads\/2020\/08\/cluster-portainer-viz-735x1024.png 735w, https:\/\/lab.rapternet.us\/wp-content\/uploads\/2020\/08\/cluster-portainer-viz-215x300.png 215w, https:\/\/lab.rapternet.us\/wp-content\/uploads\/2020\/08\/cluster-portainer-viz-768x1071.png 768w, https:\/\/lab.rapternet.us\/wp-content\/uploads\/2020\/08\/cluster-portainer-viz-1102x1536.png 1102w, https:\/\/lab.rapternet.us\/wp-content\/uploads\/2020\/08\/cluster-portainer-viz.png 1274w\" sizes=\"auto, (max-width: 735px) 100vw, 735px\" \/><\/figure>\n","protected":false},"excerpt":{"rendered":"<p>Why Why not? In reality, I always wanted to play with clustering, originally with proxmox and ceph, but I never had enough hardware to do so. I do however have a proxmox node with enough ram that I can host multiple lightweight nodes. Docker swarm is lightweight enough that I can virtualize the entire cluster &hellip; <\/p>\n<p class=\"link-more\"><a href=\"https:\/\/lab.rapternet.us\/?p=281\" class=\"more-link\">Continue reading<span class=\"screen-reader-text\"> &#8220;Virtualized Docker Swarm&#8221;<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[4],"tags":[32,13,34,22],"class_list":["post-281","post","type-post","status-publish","format-standard","hentry","category-how-to","tag-clusters","tag-docker","tag-gluster","tag-services"],"_links":{"self":[{"href":"https:\/\/lab.rapternet.us\/index.php?rest_route=\/wp\/v2\/posts\/281","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/lab.rapternet.us\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/lab.rapternet.us\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/lab.rapternet.us\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/lab.rapternet.us\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=281"}],"version-history":[{"count":7,"href":"https:\/\/lab.rapternet.us\/index.php?rest_route=\/wp\/v2\/posts\/281\/revisions"}],"predecessor-version":[{"id":339,"href":"https:\/\/lab.rapternet.us\/index.php?rest_route=\/wp\/v2\/posts\/281\/revisions\/339"}],"wp:attachment":[{"href":"https:\/\/lab.rapternet.us\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=281"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/lab.rapternet.us\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=281"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/lab.rapternet.us\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=281"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}