clusters – Systems Lab

October 1, 2020August 29, 2020

Disabling the Aggressive Inspircd Health Check

One of the first application stacks I went to install and setup on my new Raspberry Pi docker cluster was Inspircd/qwebirc/anope. This stack was running originally on a raspberry pi 1b (256MB RAM version). I wanted to move this off the pi1 since it was out of date and would need a complete reinstall to be back to full patch status. However shortly after getting it running in my swarm, I ran into issues.

The IRC server would restart every few hours, sometimes it would restart every 10 minutes or so. I deemed that as unacceptable on my basic setup even for just using it in development of IRC bots.

September 10, 2020September 12, 2020

Connecting Inspircd and Anope on Docker-Swarm

I encountered some issues when connecting Anope and Inspircd on the docker swarm. When running them on a single node using docker-compose, the services were able to connect just fine to Inspircd, however running in a docker swarm, there were issues in how Inspircd was filtering IPs in the default services XML block.

In order for Inspircd and Anope to talk I had to comment out some of the link block for the services. The allowmask and expected IP address didn’t quite work as intended inside the docker swarm cluster. Since the containers use a local network available to only them, I’m not worried about a rogue services server getting in. This IRC server is also primarily used for developing and testing the IRC bots I have written over the years, so its not a core part of my infrastructure. Below shows the link block that I modified from the default included in the inspircd docker container setup.

August 29, 2020August 29, 2020

Jenkins Docker Revisit

After my initial jenkins setup, I thought my system would be good to go for a long time, however I encountered a problem with permissions after my docker cluster reboot. After all my nodes were back up, and jenkins was running, it could no longer access the docker.sock that it used to handle building and pushing containers. I tried a few things, rebuilding the container, updating it, changing some groups, and found quite a few threads on the topic. Some people had chmod’d the docker.sock to 777 (BAD) or had given jenkins root (ALSO BAD). I ended up finding the solution in using a specific entrypoint script that would determine the group to add to the jenkins user, then launch jenkins using the jenkins user from root.

Most of my additions are from sudo-bmitch’s jenkins-docker repository on GitHub. These include the dockerfile changes and the entrypoint.sh script (as well as the healthcheck mentioned later on).

August 29, 2020

Docker Cluster Reboot

Due to unforeseen events, I ended up having to shutdown all of my servers. Due to this, my docker VM cluster ended up getting its first chance to reboot the entire stack of all nodes. This ended up showing some problems in my configuration sadly, as well as a docker issue I had previously encountered in my raspberry pi cluster.

August 27, 2020August 23, 2020

Raspberry Pi Docker Cluster

I’ve always wanted to experiment with clustering technologies, I tried setting up a kubernetes cluster however that ended in failure. For this next experiment, I went with something simpler to deal with, docker swarm. Since docker and swarm are supported on raspberry pi’s, and since i had a number of raspberry pi’s not in use, I decided to use them for the cluster.

I printed a 2U rack mount kit for raspberry pis. I felt like this would be the perfect time to make use of it. I racked up 2 raspberry pi 3B+ units with POE hats (more on that later) and went to use those for the docker cluster. I added Samsung 32GB micro sd cards for storage.

August 21, 2020August 23, 2020

Virtualized Docker Swarm

Why

Why not? In reality, I always wanted to play with clustering, originally with proxmox and ceph, but I never had enough hardware to do so. I do however have a proxmox node with enough ram that I can host multiple lightweight nodes.

Docker swarm is lightweight enough that I can virtualize the entire cluster on my single proxmox host. While this isn’t fault tolerant like a cluster across multiple nodes, it does mean I can reboot cluster nodes for kernel updates and maintain my uptime. I also am able to add additional docker swarm nodes on separate hardware if I get additional hardware, and there is the benefit of having the cluster load balance itself for which software is running on which node.

Benefits

Each node in the cluster is identical, each can be replaced by following the exact same process and while I don’t have automated deployment of new nodes, they are still closer to cattle than many of my other virtual machines. Due to the goal of replicated storage between the nodes, I should also be able to take a single node and rebuild the entire cluster if needed, since it would have the entire clusters configuration.