Jenkins Docker Revisit

After my initial jenkins setup, I thought my system would be good to go for a long time, however I encountered a problem with permissions after my docker cluster reboot. After all my nodes were back up, and jenkins was running, it could no longer access the docker.sock that it used to handle building and pushing containers. I tried a few things, rebuilding the container, updating it, changing some groups, and found quite a few threads on the topic. Some people had chmod’d the docker.sock to 777 (BAD) or had given jenkins root (ALSO BAD). I ended up finding the solution in using a specific entrypoint script that would determine the group to add to the jenkins user, then launch jenkins using the jenkins user from root.

Most of my additions are from sudo-bmitch’s jenkins-docker repository on GitHub. These include the dockerfile changes and the entrypoint.sh script (as well as the healthcheck mentioned later on).

Dockerfile

The first thing I considered doing was to add a bit to the dockerfile that would check the group that had access to the docker.sock and set the jenkins user in the container to have that group. However, since I run a 3 node docker-swarm cluster, if that ownership group has a different GID on any of the nodes, the solution may not have been a complete solution. This solution should be fine for a single node and is included below for reference.

CMD DOCKER_GID=$(stat -c '%g' /var/run/docker.sock) && \
   groupadd -for -g ${DOCKER_GID} docker && \
   usermod -aG docker jenkins && \

Instead I went with using the script by sudo-bmitch, and adding GOSU to my dockerfile to support that. The main addition here is adding GOSU to the installation, pulling in the entrypoint.sh script from my repo checkout, and updating the entrypoint to be that new script.

FROM jenkins/jenkins:lts
USER root
RUN apt-get update && \
apt-get -y install apt-transport-https \
    ca-certificates \
    curl \
    ant \
    gnupg2 \
    software-properties-common && \
curl -fsSL https://download.docker.com/linux/$(. /etc/os-release; echo "$ID")/gpg > /tmp/dkey; apt-key add /tmp/dkey && \
add-apt-repository \
    "deb [arch=amd64] https://download.docker.com/linux/$(. /etc/os-release; echo "$ID") \
    $(lsb_release -cs) \
    stable" && \
apt-get update && \
apt-get -y install docker-ce
RUN apt-get install -y docker-ce
RUN usermod -a -G docker jenkins


ARG GOSU_VERSION=1.10
RUN dpkgArch="$(dpkg --print-architecture | awk -F- '{ print $NF }')" \
 && wget -O /usr/local/bin/gosu "https://github.com/tianon/gosu/releases/download/$GOSU_VERSION/gosu-$dpkgArch" \
 && chmod +x /usr/local/bin/gosu \
 && gosu nobody true

COPY entrypoint.sh /entrypoint.sh
RUN chmod +x /entrypoint.sh
ENTRYPOINT ["/entrypoint.sh"]
HEALTHCHECK CMD curl -sSLf http://localhost:8080/login >/dev/null || exit 1

entrypoint.sh

I’ve included the entrypoint script here for reference. Its worked well so far, correcting my problem with building/pushing docker containers, and I’ve confirmed that the jenkins process is running as the jenkins user, and not root.

#!/bin/sh

# By: Brandon Mitchell <public@bmitch.net>
# License: MIT
# Source Repo: https://github.com/sudo-bmitch/jenkins-docker

set -x

# configure script to call original entrypoint
set -- tini -- /usr/local/bin/jenkins.sh "$@"

# In Prod, this may be configured with a GID already matching the container
# allowing the container to be run directly as Jenkins. In Dev, or on unknown
# environments, run the container as root to automatically correct docker
# group in container to match the docker.sock GID mounted from the host.
if [ "$(id -u)" = "0" ]; then
  # get gid of docker socket file
  SOCK_DOCKER_GID=`ls -ng /var/run/docker.sock | cut -f3 -d' '`

  # get group of docker inside container
  CUR_DOCKER_GID=`getent group docker | cut -f3 -d: || true`

  # if they don't match, adjust
  if [ ! -z "$SOCK_DOCKER_GID" -a "$SOCK_DOCKER_GID" != "$CUR_DOCKER_GID" ]; then
    groupmod -g ${SOCK_DOCKER_GID} -o docker
  fi
  if ! groups jenkins | grep -q docker; then
    usermod -aG docker jenkins
  fi
  # Add call to gosu to drop from root user to jenkins user
  # when running original entrypoint
  set -- gosu jenkins "$@"
fi

# replace the current pid 1 with original entrypoint
exec "$@"

Additional Healthcheck

This is something completely unrelated to the current goal of getting jenkins back up and running, building and deploying containers. Instead, I found in my googling a healthcheck script for the docker container and added it to mine to see how it worked. So far so good, so its shown below.

HEALTHCHECK CMD curl -sSLf http://localhost:8080/login >/dev/null || exit 1

Conclusion

This is a great way to avoid both the docker.sock problems with different GID ownership across different nodes, and also a great way to avoid security problems that simpler solutions have, while not requiring much legwork other than adding a script to load in on the dockerfile.