Taylor - Getting Docker to not suck for Development

Getting Docker to not suck for Development

3 Oct 2024
- 11 minutes to read

Too Long; Won’t Read

The short, sharp, and shiny of it all is docker containers usually run as root, so when you mount your local file system to it, it writes files as root to your machine. This bugs me and causes issues. The fix is to build your docker image with a non-root user that has your UID and GID. You can see my example in this zip file.

Unzip that and run these commands to get a working website on localhost:3001:

dx/build
dx/start
dx/exec bin/setup
dx/exec bin/run

To shut it down do the usual ^C and then run dx/stop to stop the docker containers.

The Situation

No, not him. I’ve just finished reading and working through Sustainable Dev Environments with Docker and Bash by Dave Copeland. It was a great read and has helped me reach a place where I’m now happy with docker as a development environment (so far, it’s only been a couple days). This is actually something I’ve tried to do a few times in the past, but have never found anything that worked sensibly or didn’t add a lot of friction to my development experience. Dave’s approach of using a running container that does nothing and you launch your commands via a helpful dx/exec has proven to remove nearly all of that friction.

Now, you’ll notice I said nearly. That’s because it didn’t cross one of my biggest complaints about using docker as a development environment off my list. And that complaint relates to file ownership.

Let’s take this Dockerfile for instance.

# Dockerfile

FROM debian
WORKDIR /app
CMD ["touch", "blah.txt"]

If we now run

docker build . --tag ownership-example && docker run --mount type=bind,source=.,target=/app ownership-example && ls -l

on OSX using docker desktop you’ll see a very sensible result of:

sean@Seans-MacBook-Pro ~/tmp/beep
$ docker build . --quiet --tag ownership-example && docker run --mount type=bind,source=.,target=/app ownership-example && ls -l
sha256:99bbc5d3cb705b81c4dd7a15eec755e1167dc2093e47547f9565d2c39effaf6d
total 8
-rw-r--r--@ 1 sean  staff  65  2 Oct 10:40 Dockerfile
-rw-r--r--  1 sean  staff   0  2 Oct 10:41 blah.txt

Yeah, I’ve named my work MacBook very creatively, but the main take away here is that blah.txt is owned by sean. A very sensible and expected outcome of having run a command not as a root user (right?). If we repeat this experiment on my Debian system using the standard docker install you’ll notice an interesting difference.

sean@Atom ~/tmp/beep
$ docker build . --quiet --tag ownership-example && docker run --mount type=bind,source=.,target=/app ownership-example && ls -l
sha256:9192213302d35c9b892768dfb8149fc0569c853b3ba1fe506a49de4fa7a21e19
total 4
-rw-r--r-- 1 root root  0 Oct  2 10:40 blah.txt
-rw-r--r-- 1 sean sean 65 Oct  2 10:34 Dockerfile

My Debian system is much more creatively named Atom because it used to be powered by an Intel Atom CPU. The actual take away here though is that blah.txt is owned by root????

This happens because the default user in a docker container is root. The reason this doesn’t happen on OSX is because docker desktop has to do a bunch of things to actually get your mounted folder inside the virtual machine it runs your docker container in. So I’m guessing they do some magic to prevent these sorts of things from leaking out. So what can we do about this? First, let’s establish a scenario.

The Scenario

Let’s setup a sample application with just enough complexity to be annoying. It’ll be a little sinatra application that pulls a little info out of redis. This gives us some ruby code to work with and a data store to deal with.

# Gemfile
source "https://rubygems.org"

# Webserver
gem "puma"
gem "rackup"
gem "sinatra"

# Data store
gem "redis"

# app.rb

require "sinatra"
require "redis"

set :bind, "0.0.0.0"
set :port, 3000

get "/" do
  redis = Redis.new(url: ENV.fetch("REDIS_URL", "redis://redis"))
  <<~HTML
    <h1>Hello there!</h1>
    <strong>Sample key:</strong> #{redis.get("sample-key") || "sample key not set!"
  HTML

rescue Redis::CannotConnectError
  "Can't connect to redis!\n"
end

Enter Docker

Let’s introduce a simple docker file to help us run this!

FROM debian:12

RUN apt-get update --yes && \
  apt-get install ruby --yes

RUN gem install bundler

WORKDIR /app

CMD ["sleep", "infinity"]

Pretty simple yeah? Now this simple solution will work happily if you’re using docker desktop and aren’t really fussed that Debian stable has ruby 3.1. But if you’re like me you probably want the latest ruby for all the good security fixes and what not, or maybe you’re maintaining a legacy application and need ruby 3.0. Whatever your reason it doesn’t matter. Now you might say:

Why not just use FROM ruby:3.3
– You, probably

Well, then what if you’re working on a rails application and need a specific version of node? You can’t just also have FROM node:latest (not without multi stage builds, which won’t be very nice for development).

You’ll also be stuck running everything as root and polluting your environment with root owned files (if you aren’t using docker desktop).

I had a crack at getting ruby and nodejs installed via apt-get, then configuring them to work as a non-root user, then booting our application. I got very close, but it was clear this was awkward and I wasn’t able to specify specific version of ruby and nodejs (easily).

That’s when I realised what I was doing wrong, I was treating this like a production docker image and not like a developer machine. So I slapped asdf in the image and came up with this docker file.

# Dockerfile.dev

FROM debian:12

ARG USER_ID=1000
ARG GROUP_ID=1000

ENV ASDF_VERSION=v0.14.1
ENV RUBY_VERSION=3.3.5
ENV PATH="/home/me/.asdf/shims:/home/me/.asdf/bin:$PATH"

RUN apt-get update --quiet --yes

# Install dependencies for asdf
RUN apt-get install --quiet --yes curl git

# Install dependencies for ruby compilation
RUN apt-get install --quiet --yes build-essential ruby-build libffi-dev libyaml-dev

WORKDIR /home/me/app

# Setup the group and user to match your local system
RUN groupadd --gid ${GROUP_ID} mygroup && \
  useradd -ms /bin/bash me --uid ${USER_ID} --gid ${GROUP_ID} && \
  chown -R me /home/me

USER me

RUN echo "source /home/me/.asdf/asdf.sh" >> /home/me/.bashrc && \
  echo "source /home/me/.asdf/completions/asdf.bash" >> /home/me/.bashrc

RUN git clone https://github.com/asdf-vm/asdf.git /home/me/.asdf --branch ${ASDF_VERSION}

RUN asdf plugin-add ruby && asdf install ruby ${RUBY_VERSION} && asdf global ruby ${RUBY_VERSION}

RUN gem install bundler && \
  bundle config set path vendor/bundle

CMD ["sleep", "infinity"]

You’ll probably have quickly noticed this is a bit more complicated than our previous docker files. I’ll breakdown the most important parts here and the rest I’ll leave as an exercise to the reader.

ARG USER_ID=1000
ARG GROUP_ID=1000

These two lines setup build arguments, we’ll use these to feed in your local users UID and GID.

# Setup the group and user to match your local system
RUN groupadd --gid ${GROUP_ID} mygroup && \
  useradd -ms /bin/bash me --uid ${USER_ID} --gid ${GROUP_ID} && \
  chown -R me /home/me

These bash commands add the matching group and user to the docker image, so when we run things in the container we get sensible looking results. It also allows us to use our home directory which is very useful for a lot of development tooling.

WORKDIR /home/me/app

I’m drawing attention to this line because this is where we’ll mount the code from our machine into the container. I’ve nested it under the docker user’s home directory as that’s how I usually work on my local machine, so I think it makes sense. It could just have easily been /mnt/code or /mount/local but that’s up to you to decide.

# dx/docker/docker-compose.dev.yml

services:
  app:
    image: ${IMAGE}
    init: true
    user: "me"
    volumes:
      - type: bind
        source: "../../"
        target: "/home/me/app"
    ports:
      - "3001:3000"

  redis:
    image: redis
    restart: unless-stopped
    volumes:
      - example-redis-data:/data

volumes:
  example-redis-data:

This is the docker compose file which tells our app container how to run in the background and also starts our redis server. You’ll also note the user: "me" line, this instructs everything we do in that container to default to using the me user which mimics the user on our machine.

The last bit of magic is this shell code:

# dx/shared.lib.sh
#...
if [ "$(uname)" == 'Darwin' ]; then
  log "Using 1000:1000 since we're on OSX"
  export USER_ID=1000
  export GROUP_ID=1000
else
  # Used so the app runs as your user on Linux
  export USER_ID=$(id -u)
  export GROUP_ID=$(id -g)
  log "Using ${USER_ID}:${GROUP_ID} since we're on Linux"
fi
#...

You’ll note here that we force 1000:1000 on OSX, and that’s for two reasons. Linux doesn’t usually run non-special users as a UID under 1000, and since you’re almost certainly using docker desktop, this will just work fine anyway.

With all of these, you can now use the dx scripts to boot up the application! So grab this zip file, extract it to a directory and run:

dx/build
dx/start
dx/exec bin/setup
dx/exec bin/run

And you should have a web application running on localhost:3001 that can contact its own redis. When you’re done with that, press ^C and run dx/stop to stop the containers in the background.

I strongly encourage you to check out all the files under dx/ and to check out Dave’s great book Sustainable Dev Environments with Docker and Bash if you want to understand more about these files.

Final Notes

I’ve been using this setup for a couple of days now on my personal projects and have found it working quite well. I’m sure this isn’t a perfect solution by any means, I’m sure there will be issues I hit that make me tear my hair out. But for now, when I’m working on simple rails applications that just need postgres and redis it works fantastically. I do have it working with my feature tests using Firefox inside the container, I haven’t noticed any speed differences either.

I also haven’t had to deal with sensitive secrets, things like AWS API keys. I suspect you can just not commit the dx/docker/docker-compose.env file. Actually now that I’ve looked into it you can just specify multiple env files so I’d probably just have an uncommitted one that you populate before you run dx/start.