Pushing big docker images more quickly using docker buildx and your EKS cluster

Building containers is easy when your applications are small. I’m usually doing projects in Golang, but at work we’re a monolithic Django application and after we install all our system dependencies and application dependencies, we end up with a 4.5GB container. It takes a while to build and even longer to push the container. Because of that, I started re-reading any literature that I could find about improving your Dockerfile usage.

It comes as no surprise that the first thing everyone tells you to do is to try using multi-stage builds. It makes sense. In a language like Go, you can download all of your dependencies in one stage and then copy your binary out to the stage that’ll actually be your container image. Unfortunately, Python ecosystems don’t work quite like that. You can install everything into your virtual environments, but there isn’t quite the same concept as a single binary. There’s a project called pex which does something close, but building a pex artifact just reduces everything into a single file which can help but doesn’t actually reduce the size of anything. This is kind of an over simplification but the pex format can be thought of as simply zipping up the virtual environment. You still include everything.

One of the options that I came across in my research is to use the kubernetes driver for docker buildx. You can read more about it in this blog post but simply put, you can schedule a remote docker backend to execute your docker commands. The kubernetes driver will do everything for you to schedule the agent in your cluster without having to deal with a bunch of YAML or helm charts. This is really promising and spoiler alert, it’ll come in clutch in a few paragraphs.

So let’s take a quick step back. What’s making this Django image so large? It’s a combination of system dependencies and application dependencies. How can we reduce things? I think the first thing we need to do is simulate the environment so that we start measuring things.

I’ve created a very simple GitHub repository at abatilo/large-python-container for me to experiment with what knobs we can mess with for improving the build and push times. In order to create a gigantic docker image, I install allennlp which is a natural language processing and machine learning toolkit that installs a bunch of other dependencies. Really, we don’t care what’s being installed. We just need a big container. First a quick requirements.in file, then a quick pip-compile and you get a big requirements.txt. Let’s write a minimal Dockerfile with an ubuntu:20.04 base and we’re off to the races. allennlp installs a huge list of other dependencies and so just by installing the single dependency and python3.9 into the docker container, we get an image with a grand size of:

⇒  docker build -q --no-cache https://github.com/abatilo/large-python-container.git\#main
⇒  docker images
REPOSITORY   TAG       IMAGE ID       CREATED          SIZE
<none>       <none>    7e1daa28222d   36 seconds ago   4.13GB
ubuntu       20.04     1318b700e415   3 weeks ago      72.8MB

Alrighty then. An easy 4GB container for us to start experimenting with. So, what can we do about it? We have system dependencies and our pip dependencies. Our container is incredibly minimal and there’s not anything we can exclude from the build. We’re installing python and installing allennlp. So what are our other knobs?

The first thing that I was curious about was to see whether or not additional compute would help at all. My computer has 8 cores, 16 threads. So let’s build the container locally and see if I limit the --cpu-shares for the build if there’s a difference. We’ll docker pull ubuntu:20.04 to start so that the builds we’re timing is timing exclusively the build process.

There’s a required minimum --cpu-shares of 2, so let’s start there:

⇒  time docker build -q --cpu-shares 2 --no-cache https://github.com/abatilo/large-python-container.git\#main
sha256:7175958b27433301b1d3685163b39d9025876a8a88cf9d852311e21c7eb0cafb
docker build -q --cpu-shares 2 --no-cache   0.16s user 0.04s system 0% cpu 3:27.71 total

And now with more compute?

⇒  time docker build -q --cpu-shares 16 --no-cache https://github.com/abatilo/large-python-container.git\#main
sha256:ec7faa045fc8317d4230da8a537d4a6ea6d0568b3d3a9522a3feeb104015597e
docker build -q --cpu-shares 16 --no-cache   0.14s user 0.03s system 0% cpu 3:23.37 total

So even going from 2 CPUs to 16 CPUs, the build time is almost exactly the same. This actually surprised me when I ran the test but then when I thought more about it, it made sense. Neither apt-get nor pip will do any of their operations in parallel, so we’re not going to be CPU bound at all. Well, this isn’t exactly boding well. There are no dependencies for us to remove to shrink the image size and adding more compute doesn’t help. If we had a bunch of stages in this docker container that we could actually build in parallel, then more compute could absolutely help in reducing build time, but this sample Dockerfile is a single stage and is largely representative of what I care about. Even in the case of things like a Go binary build, the second stage is serially dependent on the binary being built in the first stage. So, we can’t build the container any faster, and we can’t shrink the container at all. What do we have left? There’s just I/O. Pushing the container is also slow because of how much container there is to push.

GitHub Actions is my choice of automation/CI platform these days. I’ve been using GHA since private beta and really enjoy the platform. GHA offers hosted runners which are themselves running in Azure. This means that if we’re using GHA to build and push our containers, we’re pushing the images from Azure to our container registry of choice. You might have guessed based on the title of this post that I use EKS. I run an EKS cluster and host images in Amazon’s ECR. So the big question on my mind now is, if I’m already running compute in AWS, what if I push my containers from AWS? Would that make the push speed any faster? What are my options for pushing the images from within AWS if I’m using EKS?

For this experiment, I want to test 4 different configurations and compare them.

1. We’ll use a hosted GitHub Action runner and use the default docker container driver for docker buildx. code

2. We’ll use a hosted GitHub Action runner but use the kubernetes docker buildx driver and schedule the buildx agent to run inside an EKS cluster. code

3. We’ll use the default docker container driver but we’ll run our own custom GitHub Action runner that’s scheduled in an EKS cluster. code

4. We’ll combine options 2 and 3. We’ll use the kubernetes driver and execute it from a custom registered GitHub Action runner. code

GitHub Action hosted runners are provisioned with 2 CPUs and 7GB of memory.

The buildx agents are configured with the following driver options:

      - name: Create buildx daemon on EKS cluster
        uses: docker/setup-buildx-action@v1
        with:
          driver: kubernetes
          driver-opts: |
            replicas=1
            namespace=buildx
            requests.cpu=2
            requests.memory=7Gi
            limits.cpu=2
            requests.memory=7Gi

The custom runners were registered with the actions-runner-controller project with the following RunnerDeployment defined in my terraform:

resource "kubernetes_manifest" "runner" {
  manifest = {
    apiVersion = "actions.summerwind.dev/v1alpha1"
    kind       = "RunnerDeployment"
    metadata = {
      "name"      = "runner"
      "namespace" = "buildx"
    }
    spec = {
      replicas = 2
      template = {
        spec = {
          repository = "abatilo/large-python-container"
          labels     = ["custom-runner"]
          resources = {
            requests = {
              cpu    = "2"
              memory = "7Gi"
            }
            limits = {
              cpu    = "2"
              memory = "7Gi"
            }
          }
        }
      }
    }
  }
}

The EKS cluster itself is provisioned to use c5n.2xlarge instances for this experiment.

So what happened when I ran all this?

Runner type | Driver type | Time elapsed
----------- | ----------- | ------------
Hosted      | default     | 9m16s
Hosted      | k8s         | 7m34s
Custom      | default     | 7m27s
Custom      | k8s         | 7m17s

I ran this test a few different times and it always resulted in roughly the same results. There was approximately a ~28% improvement in how long it takes to push the container. The build times of the container themselves were pretty consistent which is expected. It didn’t matter which method I used for build and push the container from within AWS, but it was always faster than using the default GHA runner. Frankly, I think the results make a lot of sense. Out of curiosity, I did also try the tests using just c5.2xlarge instead of the c5n variant. There wasn’t any noticeable difference in the timing.

Using the Kubernetes buildx driver was extremely simple to setup and got all the benefits, so I definitely plan on pushing that configuration at work when it’s appropriate. This way, you don’t have to worry about configuring and maintaining the actions-runner-controller. That being said, you’d be double paying for compute. You’re paying for the GHA minutes to run the container that’s just delegating the build to compute that you’re paying for in the EKS cluster.

So we’ve sped up the push time with more locality. What else could we do? What about caching the docker images? These builds were actually configured for doing type=registry based caching with ECR configured with mode=max which would push every layer of every stage to ECR. Turns out that as of writing this (August 2021) ECR doesn’t support the required metadata for this type of caching. That being said, using the type=inline caching did work. And maybe surprisingly, this made all 4 build strategies take about the same amount of time when it came to doing builds that didn’t change any dependencies.

Thanks for reading. If I find other ways to speed things up, maybe I’ll remember to write about it.