How to use S3 for your docker layer cache

ECR doesn't support cache manifests, but S3 does!

As of May 2022, the buildkit backend for building containers has experimentally supported using S3 as a backend cache. For over 2 years, there’s been no other way to store max mode container manifests in AWS, as ECR still doesn’t support cache manifests. On the bright side, in October 2022, we finally started getting some communications from AWS about supporting this, but since there still isn’t anything implemented, we need to look at alternatives.

Since the S3 backend is experimental, it’s still has almost no practical documentation for how to use it, but in this post we’ll go through how I was able to make it work and the benefits that I saw from implementing the cache.

Setting up our example project

Since this feature relies on buildkit, make sure that you have docker buildx installed, which is the integration between the docker CLI and the tool agnostic buildkit backend.

By default, docker images are built with what’s called min mode caching. You can drastically improve caching behaviors by using max mode caching. From the official buildx documentation, the difference is described as:

In min cache mode (the default), only layers that are exported into the resulting image are cached, while in max cache mode, all layers are cached, even those of intermediate steps.

While min cache is typically smaller (which speeds up import/export times, and reduces storage costs), max cache is more likely to get more cache hits. Depending on the complexity and location of your build, you should experiment with both parameters to find the results that work best for you.

By this point, using multi-stage builds is a well documented best practice for having the smallest images. Smaller images are faster to push and pull which can make your infrastructure much more agile and elastic. Let’s simulate an example of what might be a very common use case of a multi-stage build: installing your node dependencies in one stage, and keeping your build artifacts in another. You don’t need all of your node_modules in your final image for the app to run, so they don’t get pushed/cached anywhere if you use a min mode cache. Let’s check out a popular nextjs app to do our experiments on. carbon is a project that lets you create beautiful images of source code.

⇒  gh repo clone carbon-app/carbon
⇒  cd carbon

Let’s add in a .dockerignore file to it.

⇒  cat << EOF > .dockerignore
node_modules
.next
EOF

This will ignore any local folders that get built that would taint the clean build environment of a Dockerfile.

And now let’s create a multi-stage Dockerfile to build

ARG TAG=18.10.0-alpine

# Install production dependencies only
FROM node:${TAG} as dependencies
WORKDIR /app
COPY package.json yarn.lock ./
RUN yarn install --production=true --frozen-lockfile

# Install dev dependencies like the
# typescript compiler so that we can build the app
FROM node:${TAG} as builder
WORKDIR /app
COPY package.json yarn.lock ./
RUN yarn install --frozen-lockfile
COPY . .
RUN yarn build

# Our final image that will be used in production
FROM node:${TAG}
COPY --from=dependencies /app /app
COPY --from=builder /app/.next /app/.next
COPY --from=builder /app/public /app/public
WORKDIR /app
CMD ["yarn", "run", "start"]

Here we have 3 stages. The first one will install only the dependencies that we need in production. The second will include devDependencies for things like linting, testing, typechecking, etc. The last stage will be the built app with only the production dependencies.

What’s the difference look like right now?

⇒  docker buildx build --load --no-cache -f Dockerfile -t carbon .
...

⇒  docker buildx build --load --no-cache -f Dockerfile --target builder -t carbon-builder .
...

⇒  docker images
REPOSITORY                 TAG              IMAGE ID       CREATED          SIZE
carbon-builder             latest           9edfbea56eef   2 minutes ago    2.56GB
carbon                     latest           38e9ab65ff02   7 minutes ago    670MB

So if you include all of the devDependencies in this image, that’s about 2GB more bloat that could end up in your final image that you try to deploy. That’s going to take much longer for your nodes to pull and all thanks to code sitting in the container that has nothing to do with running your actual application. However, if you don’t include all that in your min mode cache build, you’ll have to re-download everything with `yarn install` every single time you build this container. Remember, min mode caching only keeps whatever’s in the last layer, which wouldn’t include the devDependencies in this optimized setup. So how can we cache different stages to keep our build times fast, but keep the small image?

Configuring the S3 remote cache

Firstly, you might be wondering, why would I want to use the S3 remote cache? After all, there’s already registry caches or the GitHub Actions cache. Honestly, these work wonders already! If you use something like the GitHub Container Registry that already supports cache manifests, then you should just use that. Even if your images are in ECR, that doesn’t mean the S3 cache will be faster, since your builders, like the default GitHub Actions, run in GitHub’s/Azure’s cloud. If you build your images there, then you might as well use the GitHub Actions cache, to pull and push the cache layers more quickly, even if your final images get pushed to ECR.

However, if you’re like me, you might be interested in saving 30% of your time by building and pushing your big docker images by running your builders from within AWS itself. If this is the case, then your custom GitHub Action runners or your remote buildx agents might actually be in AWS, and then pushing and pulling your cached layers to GitHub’s data centers, which has a non-trivial delay associated with it. So let’s use the S3 cache to speed up those builds!

---

Let’s create some remote buildx agents that run in AWS just like was outlined in this previous newsletter post that I just mentioned about saving time. This feature hasn’t made it into a release branch yet for the moby/buildkit docker image, so when we create a buildx agent, we’ll need to specify that we want to use the image built on the master branch, hence why you see me set image=moby/buildkit:master

⇒  docker buildx create \
--name=s3-cache \
--bootstrap \
--use \
--driver=kubernetes \
--driver-opt=requests.cpu=2,requests.memory=7Gi,limits.memory=7Gi,replicas=1,image=moby/buildkit:master

I’m using a remote agent with the Kubernetes driver to run a remote buildx agent that’s located in the us-west-2 region of AWS. This agent has 2CPUs and 7Gi of memory just like the standard GitHub Action runners.

So first, let’s run the build without any caching and see how long it takes:

⇒  time docker buildx build .
...
docker buildx build --no-cache .  1.26s user 0.55s system 1% cpu 1:46.36 total

For good measure for this next test, I’m going to re-create the builder entirely before running the next test, to make sure that we doing have any on-disk caching happening. I’ve already ran the following command once in order to seed S3 with the cache, but the node doesn’t have anything on disk.

⇒  time docker buildx build \
--cache-from "type=s3,region=us-west-2,bucket=abatilo,access_key_id=$(aws --profile s3-cache configure get aws_access_key_id),secret_access_key=$(aws --profile s3-cache configure get aws_secret_access_key),prefix=buildx/carbon/" \
--cache-to "type=s3,region=us-west-2,bucket=abatilo,access_key_id=$(aws --profile s3-cache configure get aws_access_key_id),secret_access_key=$(aws --profile s3-cache configure get aws_secret_access_key),prefix=buildx/carbon/,mode=max" .
...
docker buildx build --cache-from  --cache-to  .  1.97s user 0.50s system 21% cpu 11.738 total

So with nothing cached at all, it took 1 minute and 46 seconds. With the cache artifacts in S3 already on a brand new remote agent, the build took 11.7 seconds to download the cached artifacts and recognize there aren’t any changes. That’s almost a 10x improvement.

Note that in running the command, you need to pass in the access_key_id and the secret_access_key directly into the cache-from and cache-to configuration. If you do not specify these from the client, credentials will be looked for on the server side. There is presently no way to run a docker buildx create command to create agents with credentials on the server side. In my case, I’m using the aws configure get command to fetch the credentials from my local aws config files.

Specifying the credentials within the command was the magic sauce for me, because I kept missing the note about credentials being looked for on the server.

access_key_id, secret_access_key, and session_token, if left unspecified, are read from environment variables on the BuildKit server following the scheme for the AWS Go SDK. The environment variables are read from the server, not the Buildx client.

And with that, we’re off! If you’d like to see a real example of configuring this remote cache in a GitHub Action, I’ve made a small repo to go along with this newsletter post that demonstrates that. Although, this example does NOT include setting up a remote builder in AWS, only AWS authentication and how to pass that into the docker/build-push-action. In this case, fetching the cache from S3 actually ends up slower than just doing a raw yarn install, which is presumably because the CDN point of presence that is hosting the packages is actually closer to the GitHub Action hosted runners compared to having to fetch files from S3.

Thanks for reading! Please consider subscribing to receive future newsletter posts.