PID 1 purists crying in the club rn.

Why do I want to run multiple processes anyways

It’s pretty common for multi-node GPU training frameworks to communicate over something like ssh. If you try to do multi-node training with DeepSpeed, you need to configure a hostfile and then configure passwordless SSH between all of the nodes. OpenMPI does the same.

In non-containerized environments, you might use a process manager like systemd or supervisord. Docker actually documents examples of using supervisord for managing multiple processes. Which, for all intents and purposes, does exactly what we want, EXCEPT an organic way to exit when a process exits. Process managers tend to want to restart any process that have exited. Even with supervisord, you can configure autorestart = false but supervisord itself will not exit, meaning that your container itself will not exit.

Why does exiting matter

ML training jobs are not web services. A web service is intended to run forever, and will only be shutdown when it needs to be replaced. That makes sense. In theory, your website should live forever. ML training jobs are long running batch jobs. It’s fully expected that they will eventually finish running. They might run for hours, days, weeks, months or longer, but eventually you’ve gone through all of your training data and you want your process to exit. The function will eventually exit. How do we do this in a container?

Subscribe now

s6 process supervision

S6 describes itself as:

s6 is a collection of utilities revolving around process supervision and management, logging, and system initialization.

Fascinating.

On top of this, I’ve been recently made aware about s6-overlay which describes itself as:

s6-overlay is an easy-to-install (just extract a tarball or two!) set of scripts and utilities allowing you to use existing Docker images while using s6 as a pid 1 for your container and process supervisor for your services.

Fascinating… again. This would give us exactly what we want. The ability to run sshd as a long running process while still running our own application. Process signals will be managed for us, all in a relatively straightforward process manager.

In the rest of this post, I’m going to walkthrough a minimal Dockerfile that uses s6-overlay and explain the pieces of it, as I understand them.

In order to make the Dockerfile easily reproducible, we’re going to embed the configuration directly into the Dockerfile so that we don’t have to worry about setting up other files locally.

FROM ubuntu:22.04

# Install sshd and xz to unzip the s6-overlay tarball
RUN <<EOF
apt-get update;
apt-get install -yq openssh-server xz-utils;
EOF

# Install s6-overlay
ADD https://github.com/just-containers/s6-overlay/releases/download/v3.1.6.2/s6-overlay-noarch.tar.xz /tmp
RUN tar -C / -Jxpf /tmp/s6-overlay-noarch.tar.xz
ADD https://github.com/just-containers/s6-overlay/releases/download/v3.1.6.2/s6-overlay-x86_64.tar.xz /tmp
RUN tar -C / -Jxpf /tmp/s6-overlay-x86_64.tar.xz

# Define sshd as a long running s6 service
COPY <<EOF /etc/s6-overlay/s6-rc.d/sshd/type
longrun
EOF

# Define entrypoint for sshd
# Create sshd's required directory
# Daemon applications tend to log to stderr, so redirect it to stdout
# Start sshd in the foreground
COPY --chmod=700 <<EOF /etc/s6-overlay/s6-rc.d/sshd/run
#!/bin/sh
mkdir -p /var/run/sshd
/usr/sbin/sshd -D -e
EOF

# Register sshd as a service for s6 to manage
RUN touch /etc/s6-overlay/s6-rc.d/user/contents.d/sshd

# Copy my ssh public keys to the container
ADD https://github.com/abatilo.keys /root/.ssh/authorized_keys

# Define training as a one shot s6 service
WORKDIR /app
COPY --chmod=700 <<EOF /app/training.sh
#!/bin/sh
echo "Training started"
sleep 5
echo "Training finished"
EOF

ENTRYPOINT ["/init"]
CMD ["/app/training.sh"]

Let’s break it down by section:

FROM ubuntu:22.04

# Install sshd and xz to unzip the s6-overlay tarball
RUN <<EOF
apt-get update;
apt-get install -yq openssh-server xz-utils;
EOF

# Install s6-overlay
ADD https://github.com/just-containers/s6-overlay/releases/download/v3.1.6.2/s6-overlay-noarch.tar.xz /tmp
RUN tar -C / -Jxpf /tmp/s6-overlay-noarch.tar.xz
ADD https://github.com/just-containers/s6-overlay/releases/download/v3.1.6.2/s6-overlay-x86_64.tar.xz /tmp
RUN tar -C / -Jxpf /tmp/s6-overlay-x86_64.tar.xz

Installing s6-overlay is incredibly easy. I think it’s relatively unknown that Docker’s ADD directive has the ability to download from the internet, so you don’t even need to have curl or wget installed. ADD will even unxz for you, and so all you need to do from that point is to untar the s6 utilities into the rest of the filesystem. ezpz.

# Define sshd as a long running s6 service
COPY <<EOF /etc/s6-overlay/s6-rc.d/sshd/type
longrun
EOF

# Define entrypoint for sshd
# Create sshd's required directory
# Daemon applications tend to log to stderr, so redirect it to stdout
# Start sshd in the foreground
COPY --chmod=700 <<EOF /etc/s6-overlay/s6-rc.d/sshd/run
#!/bin/sh
mkdir -p /var/run/sshd
/usr/sbin/sshd -D -e
EOF

# Register sshd as a service for s6 to manage
RUN touch /etc/s6-overlay/s6-rc.d/user/contents.d/sshd

Okay, here’s the meat of using s6. The /etc/s6-overlay/s6-rc.d directory is where you can specify all of your services. By creating the sshd directory, that’s essentially declaring that you have a service that you want to run. The first file here is a file named type where you can define either a longrun service or a oneshot service. For sshd, that’s a long running service that I expect to never need to exit. Next, we have to create this run file in the sshd directory. This shell script is creating the required directory for sshd, and then runs sshd NOT as a daemon (via the -D flag, which I think feels backwards). The last line here is to create an empty file in /etc/s6-overlay/s6-rc.d/user/contents.d/ which is how you actually register what services you want s6 to execute. The name of the file needs to match the name you used in the service configuration.

# Copy my ssh public keys to the container
ADD https://github.com/abatilo.keys /root/.ssh/authorized_keys

# Define training as a one shot s6 service
WORKDIR /app
COPY --chmod=700 <<EOF /app/training.sh
#!/bin/sh
echo "Training started"
sleep 5
echo "Training finished"
EOF

ENTRYPOINT ["/init"]
CMD ["/app/training.sh"]

Lastly, here’s my quick way for me to add some public keys to my server by fetching them from GitHub. and then here’s my mock bash script that would initiate the actual training code.

The last section here is perhaps the most fascinating. We’re combining both ENTRYPOINT and CMD. The ENTRYPOINT invokes s6 itself with /init and then setting the CMD let’s us specify the entrypoint to our main service. Doing this means that we will actually instantiate s6 as our PID 1, and s6 will take the CMD as the service that we want to run. S6 will instantiate all of the services that we’ve defined, which in our case is only sshd, and once that’s running, will instantiate our main service. Once the training process exits, s6 will notice this and will send shutdown signals to all other services under its management, allowing for a clean exit. S6 will use the exit code of our main service as its overall exit code, so if something fails unexpectedly in the training process, we’ll be able to catch it in our upstream container orchestration layer.

Try it yourself with this Dockerfile:

docker build -t s6 https://gist.githubusercontent.com/abatilo/2b0c99d8536476f0219de2b8cc038870/raw/c942a1269d8c9a18e70e540c79a3a26750f56352/Dockerfile && docker run -p 2222:22 --rm -it s6

Thanks for reading.

Subscribe now

I tweeted the other day asking people if they’d be interested in learning how I’m fine tuning models for $0.03 per million tokens, and at a whopping 7 likes, it’s my most liked tweet of all time. So here I am.

This isn’t a tutorial, so don’t expect a step by step break down to do everything that I did. Although, if you’d be interested in that, let me know! I’ve been thinking about trying to do an online course or video cast or something walking you through actually setting up a Kubernetes cluster for this. For now, I’ll keep it high level with all of the infra and underlying pieces that I’m actually using.

How do I get my GPUs?

Let’s start with the actual infrastructure that’s being used then, shall we? I’m still running my very same AWS Elastic Kubernetes cluster that I’ve talked about before. But since that post, some things have stayed the same and some things have changed. I use Karpenter.sh as my cluster auto scaler, which will provision nodes for me. With that installed to the cluster, I use the following EC2NodeClass and NodePool definitions below.

Note: My EKS cluster is named “red” and my GPU node group is named “green”.

apiVersion: karpenter.k8s.aws/v1beta1
kind: EC2NodeClass
metadata:
  name: green-gpu
spec:
  amiFamily: AL2
  blockDeviceMappings:
  - deviceName: /dev/xvda
    ebs:
      encrypted: true
      volumeSize: 100Gi
      volumeType: gp3
  role: green-eks-node-group-20230615015942833200000001
  securityGroupSelectorTerms:
  - tags:
      karpenter.sh/discovery: red
  subnetSelectorTerms:
  - tags:
      karpenter.sh/discovery: red
  tags:
    karpenter.sh/discovery: red
---
apiVersion: karpenter.sh/v1beta1
kind: NodePool
metadata:
  name: green-gpu
spec:
  disruption:
    consolidationPolicy: WhenUnderutilized
    expireAfter: 1h
  limits:
    cpu: 100
    memory: 200Gi
    nvidia.com/gpu: 4
  template:
    spec:
      nodeClassRef:
        name: green-gpu
      requirements:
      - key: karpenter.sh/capacity-type
        operator: In
        values:
        - spot
      - key: kubernetes.io/arch
        operator: In
        values:
        - amd64
      - key: karpenter.k8s.aws/instance-category
        operator: In
        values:
        - g
      - key: karpenter.k8s.aws/instance-generation
        operator: Gt
        values:
        - "5"
      - key: karpenter.k8s.aws/instance-gpu-count
        operator: In
        values:
        - "1"
      - key: karpenter.k8s.aws/instance-cpu
        operator: Lt
        values:
        - "64"
      - key: kubernetes.io/os
        operator: In
        values:
        - linux
      taints:
      - effect: NoSchedule
        key: nvidia.com/gpu
        value: "true"

Reminder, let me know if you’d be interested in an end to end tutorial for setting up a cluster. But for now, let me just link to the Karpenter getting started docs. The node requirement selectors are very deliberate. I’m requesting g6 family nodes as evidenced by the “instance-generation is greater” than 5 requirement which brings me Nvidia L4 GPUs. These GPUs have 24GB of GPU memory and are typically marketed more for inference, but for this broke boy, we’ll just have to make do. I choose to only provision nodes that are spot lifecycle nodes which helps with the pricing, but I’ll break that down more in a later section.

Subscribe now

What code do I run to train the models?

Selecting for L4 GPUs is very intentional, as L4 GPUs are of the Ada generation architecture. This ties in to the actual software stack that I’m using. Back in December, a good friend of mine messaged me about some new project that got posted to /r/LocalLlama. That project was called unsloth, which made extraordinary claims for improving performance of training Llama2 models. “80% faster and 50% less memory” was pretty big. To be perfectly honest, I haven’t personally measured the difference between using unsloth and not using it, but I appreciate the tenacity. Since then, I’ve had a few conversations with the technical mind behind it, @danielhanchen, and I must say, he’s a very friendly and very brilliant guy. It’s amazing what a cold email can do!

My code is based heavily on Unsloth’s own Google Colab example. Here’s my fine tuning code verbatim.

from datasets import load_dataset
from datetime import datetime
from transformers import TrainingArguments, TrainerState
from transformers.trainer_utils import get_last_checkpoint
from trl import SFTTrainer
from unsloth import FastLanguageModel
import os
import shutil
import torch

max_seq_length = 8192

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="unsloth/llama-3-8b-bnb-4bit",
    max_seq_length=max_seq_length,
    dtype=None,
    load_in_4bit=True,
)

model = FastLanguageModel.get_peft_model(
    model,
    r=16,
    target_modules=[
        "q_proj",
        "k_proj",
        "v_proj",
        "o_proj",
        "gate_proj",
        "up_proj",
        "down_proj",
    ],
    lora_alpha=16,
    lora_dropout=0,
    bias="none",
    use_gradient_checkpointing="unsloth",
    random_state=1337,
    use_rslora=False,
    loftq_config=None,
)

alpaca_prompt = """Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.

### Instruction:
{}

### Input:
{}

### Response:
{}"""

EOS_TOKEN = tokenizer.eos_token


def formatting_prompts_func(examples):
    instructions = examples["instruction"]
    inputs = examples["input"]
    outputs = examples["output"]
    texts = []
    for instruction, input, output in zip(instructions, inputs, outputs):
        text = alpaca_prompt.format(instruction, input, output) + EOS_TOKEN
        texts.append(text)
    return {
        "text": texts,
    }


dataset = load_dataset("yahma/alpaca-cleaned", split="train")
dataset = dataset.map(
    formatting_prompts_func,
    batched=True,
)

trainer = SFTTrainer(
    model=model,
    tokenizer=tokenizer,
    train_dataset=dataset,
    dataset_text_field="text",
    max_seq_length=max_seq_length,
    dataset_num_proc=2,
    packing=False,
    args=TrainingArguments(
        per_device_train_batch_size=4,
        gradient_accumulation_steps=4,
        include_num_input_tokens_seen=True,
        warmup_steps=5,
        num_train_epochs=1,
        save_steps=50,
        save_total_limit=3,
        learning_rate=2e-4,
        fp16=not torch.cuda.is_bf16_supported(),
        bf16=torch.cuda.is_bf16_supported(),
        logging_steps=5,
        optim="adamw_8bit",
        weight_decay=0.01,
        lr_scheduler_type="linear",
        seed=1337,
        output_dir="/mnt/outputs",
    ),
)


def get_checkpoints(checkpoint_dir):
    checkpoints = [
        os.path.join(checkpoint_dir, d)
        for d in os.listdir(checkpoint_dir)
        if os.path.isdir(os.path.join(checkpoint_dir, d))
    ]
    checkpoints.sort(reverse=True, key=lambda x: int(x.split("-")[-1]))
    return checkpoints


def safe_resume_training(trainer, checkpoint_dir):
    checkpoints = get_checkpoints(checkpoint_dir)
    for checkpoint in checkpoints:
        try:
            trainer_state_file = os.path.join(checkpoint, "trainer_state.json")
            TrainerState.load_from_json(trainer_state_file)
            print(f"Resuming training from {checkpoint}")
            trainer.train(resume_from_checkpoint=checkpoint)
            return
        except FileNotFoundError as e:
            print(f"Error: {e}")
            print(f"The checkpoint {checkpoint} is incomplete and will be skipped.")
            shutil.rmtree(checkpoint, ignore_errors=True)

    print("No valid checkpoint found. Starting training from scratch.")
    trainer.train()


start_time = datetime.now()
safe_resume_training(trainer, "/mnt/outputs")
print("Training took: ", datetime.now() - start_time)
trainer.save_state()
model.save_pretrained("/mnt/llama3_lora_model")

I had ChatGPT generate the code for cleaning up faulty checkpoints. Since I’m running on spot nodes, I occasionally get nodes taken away from me that don’t get to fully checkpoint, so this is my solution for robustness.

Notice that I’m using the Unsloth quantized version of the 8b. The code sample is training using an Alpaca dataset and an Alpaca instruction format. In terms of training cheaply, I don’t think that will make a difference though. People can take my recipe and do other data with it.

What do I use for file storage?

Amazon S3 isn’t a file system, it’s an object store. Which makes it the perfect file system for me to use for storing the models. I have the Mountpoint for Amazon S3 CSI driver installed on my EKS cluster, which lets me configure an S3 bucket as a PVC. I haven’t benchmarked it for I/O throughput but for the model sizes and data set sizes that I’ve been using, it’s been great so far.

So how much does it cost?

Let’s break down the costs of this setup. I’m not going to factor in the cost of the EKS cluster itself or any CPU nodes you need. I’m only going to focus on the cost of the actual GPUs. The node definitions that I shared earlier in this post are selecting only g6 instances. I allow for selecting to larger instance types just to deal with the availability of the instances themselves. Even though both a g6.xlarge up to the g6.8xlarge only have a single L4 GPU on them, having the different amounts of CPU and memory can make them more or less appetizing to other customers. Since the spot lifecycle type is always going to be excess compute capacity, I’ll take whatever I can get when I’m playing with my infrastructure.

At the g6.8xlarge upper end of the spectrum, that historically caps out at an hourly cost of about $0.22 per hour in the us-west-2 region.

When there are g6.xlarge instances available, the prices on spot instances is a whopping $0.085 an hour. That’s less than 9 cents per hour for a 24GB GPU!

Even the absolute very cheapest instance on Vast.ai is still $0.10 per hour.

And how do I get the $0.03 number that I click baited you with? Well, that’s about how much it costs to process over 1,000,000 tokens.

In my TrainingArguments, I set include_num_input_tokens_seen=True. I log every 5 steps, so I can see how long it takes to do 5 steps and how many tokens were processed over those 5 steps. Then I can extrapolate. It takes approximately 30 seconds to do 5 steps, which gets me just about ~24,512 tokens processed. That’s about ~817 tokens per second. Which comes out to about $0.02917 per 1,000,000 tokens.

Let me know if you’d be interested in tutorials for the rest of the infrastructure.

Thanks for reading.

Subscribe now

That being said, I can appreciate that if you just care to get something running, you can do that a lot more easily with GCP than with AWS. But if you just want to get something running, IMO you should be using a more managed platform. Anyways, that’s all probably for another day or another post.

Thanks terraform-google-kubernetes-engine module, I was able to get a spot node GKE cluster up and running pretty easily. On top of that, I use ArgoCD for deploying my side projects and registering another cluster as a target was also really straightforward. I had to create a Kubernetes service account and pass that token to ArgoCD, and I hate static credentials, but that’s fine for now. I think one day I might change it around to do some OIDC based “impersonations” as GCP likes to call them.

Okay, cool. I have two clusters up. Now what? Do I create a federated mesh between them? Is there something easier? But let’s start with me telling you about what I actually did first that worked super poorly.

The first thing that I did was to run the container that I use for serving https://mentallyanimated.com but in both clusters. It’s just a single static HTML page, but you can replace that with any stateless service. I learned that by default, GCP will actually install an ingress controller for you that leverages a “classic” HTTP(S) load balancer for you. Okay, simple enough. I learned about things like the BackendConfig CRD. More notably, I learned about the ManagedCertificate CRD which would request an SSL certificate. I pointed my DNS for https://mentallyanimated.com and an hour later my SSL certificate was provisioned.

My SSL certificate was provisioned… wait, how? I couldn’t find any actual documentation about the details, but I’m assuming that the GCP load balancer is doing an HTTP01 ACME challenge and serving the validation HTTP request automatically for me. I couldn’t actually find any way to forcefully make the ManagedCertificate CRD do a DNS validation for the SSL certificate. Little did I know that this would bite me later, but whatever. In the moment, I was excited. I had my first application scheduled in GKE.

In order to do cross cloud, I ended up doing the most simple thing I could think of. I did 50/50 weighted DNS. Essentially, this is what I did:

resource "aws_route53_record" "gcp_www" {
  weighted_routing_policy {
    weight = 10
  }
  # ...everything else
}

resource "aws_route53_record" "aws_www" {
  weighted_routing_policy {
    weight = 10
  }
  # ...everything else
}

And it actually worked beautifully. I was now sometimes serving my page from AWS and sometimes from GCP. I was multi-cloud. There was just one problem. I had forgotten to add the alternative subject for my ManagedCertificate for the apex domain. https://mentallyanimated.com with no subdomain. And this is where I started to actually notice problems.

Firstly, it doesn’t seem like GCP technically supported that. Even though I was doing an A record from the domain to a static global IP address that I provisioned through GCP, the SSL certificate would just *never* provision. I waited days to see if that would help. I think because I was serving everything from two clouds, the HTTP01 challenge that I believe is happening behind the scenes was never consistently resolving, because of the DNS load balancing. So I gave up on that.

Cloudflare tunnels

Several years ago, an old mentor of mine told me about how he had successfully migrated to having completely and utterly private Kubernetes clusters, including the control plane. For whatever reason, my brain remembered that conversation and remembered that he had done so by using Cloudflare for ingress. Sure enough, there’s actually a Cloudflare ingress controller that leverages their tunnels for creating a connection from your environment into their environment, all through an egress only connection. The connection comes from your environment and are authenticated into their environment. Unfortunately, the ingress controller is super outdated. I started looking at their docs and found that they actually have documentation about exposing applications in Kubernetes, exactly like I wanted.

I reached out to the same mentor and asked if he was using the ingress controller or if he went another route. He said that he runs the cloudflared tunnel service as its own Deployment in his clusters and forward traffic from there to a gateway/reverse proxy. I had an extremely similar but different idea. documentation about exposing applications in KubernetesWhat if I run cloudflared as a sidecar to my ingress controller? I tried it, and it worked phenomenally, and all it took was a few lines of terraform to make it work. All I had to do was define a cloudflare_tunnel resource, a cloudflare_tunnel_config, and then add my additional sidecar container. It basically all just looks like this:

resource "cloudflare_tunnel" "mentallyanimated" {
  account_id = local.cloudflare_account_id
  name       = "mentallyanimated"
  secret     = data.aws_ssm_parameter.cloudflare_mentallyanimated_tunnel_secret.value
}

resource "cloudflare_tunnel_config" "traefik" {
  account_id = local.cloudflare_account_id
  tunnel_id  = cloudflare_tunnel.mentallyanimated.id

  config {
    ingress_rule {
      service = "http://localhost:8000"
    }
  }
}

resource "helm_release" "traefik" {
  # ...everything else
  values = [
    <<EOF
deployment
  additionalContainers:
  - name: cloudflared
    image: cloudflare/cloudflared:latest
    args:
    - tunnel
    - run
    - --token
    - ${cloudflare_tunnel.mentallyanimated.tunnel_token}
EOF
  ]
  depends_on = [module.traefik_crds]
}

Since cloudflared and traefik are in the same pod, all I have to do is have cloudflared forward traffic to http://localhost:8000 where traefik is listening. Traefik is already my ingress controller and will redirect traffic to my other applications based on the Host header. Having cloudflared as a sidecar means that when traefik scales out, so will cloudflared.

The best part is that since I’m using traefik as my ingress controller, it’s already cloud agnostic. I don’t need to expose it using a cloud specific load balancer anymore so I can get rid of my NLB and I can disable the GKE default ingress controller. My traefik setup in both clusters can be exactly the same and Cloudflare will do the round robin load balancing across different clouds. It’s “that easy”.

This solves the problem of doing stateless applications. I’ll throw “how to do stateful applications” cross cloud on to my to-do list. For now, this is perfect.

Recently, a friend of mine just started the Sillycon Valley newsletter where he shares what he learns about getting into AI and machine learning. He’s been playing with some image generation and asked me to run the code on my computer. With some of the memory optimization flags and running more on the CPU over my GPUs, I was able to get images to generate, but my image generation for my tower would take 2-3x as long as his laptop! I knew there had to be a better way.

I have two computers. I have an older tower that I built in ~2015 which I built to do some machine learning experiments at the time. It has two GTX 970s SLId together. Each card has 4GB of VRAM which was pretty good at the time. I could run a bunch of the open source models that were being published. Today, this is still my main computer and the one that I do all my development on. I also have a pre-built gaming computer that I bought in 2021 that has a GTX 3090 in it. I got this computer to get more into PC gaming again after years of hiatus. Hilariously, most of the time I’m only playing Super Smash Bros Melee on an emulator and I don’t need such a powerful graphics card.

Recently, all the transformer models have been getting larger and larger and the standard for a lot of these models is to use an NVIDIA A100 which has 40GB of VRAM. The larger models built by well capitalized companies are built on hundreds or thousands and maybe even tens of thousands of these. My 970s just can’t keep up anymore, so I haven’t done any ML experimenting in a while.

Since I have the 3090, I’ve thought about using that machine as my dev machine, but my brain really wants to keep it a Windows gaming PC. I think because that’s what I bought it for, that’s what I want to keep it as. You might tell me to just set it up with a dual boot, but I don’t want to do that either, because I don’t want to have to decide between the two options. For years I just left it as is and then one morning I realized that if I setup a Linux VM then I could just access the 3090 like I would access an EC2 instance.

I installed the Windows Subsystem for Linux with the default Ubuntu, and sure enough:

Without doing anything else, the WSL VM can already access the graphics cards. Perfect. The next thing that I did was configure an ssh server on the VM and then setup a port forward from the host computer to the VM. By following this tutorial, I was able to ssh right from my dev computer to the VM and thus to the graphics card.

At this point, I went and yoinked the code from my good pal, Sillycon Valley and his tutorial for running stable diffusion for doing image generation. And then to put my own flavor on the images I tested generating, I started generating some anime pictures. In real-life time, I spent a few hours experimenting with different parameters and memory and speed optimizations.

I also started doing some experimenting with some of the smaller open source chatbot models like Dolly from Databricks.

Serving the model

All it takes to run one of these models is:

import torch
from fastapi import FastAPI
from pydantic import BaseModel
from transformers import pipeline

class Req(BaseModel):
    text: str
app = FastAPI()
generate_text = pipeline(model="databricks/dolly-v2-7b", torch_dtype=torch.bfloat16, trust_remote_code=True, device_map="auto")

@app.post("/generate")
async def gen(req: Req):
    res = generate_text(req.text)
    return {'response': res[0]["generated_text"]}

This is great. Now I can send requests from my dev machine and generate all kinds of stuff with my 3090 and put it to work. But what if I want to expose things to the internet? Or just my projects? Well, I don’t want to have to expose my home network. I don’t want to do any kind of port forwarding from my router, etc. That’s where the magic of tailscale comes in. Tailscale is a vpn service built on top of WireGuard. I installed the tailscale client in the WSL VM and now it’s on my private tailnet.

Next I had to figure out how to get a tailscale client into my Kubernetes applications, so that I could have my projects access the GPU all privately and securely. Tailscale themselves have some documentation for this. All I had to do was create and set an auth token and then add the tailscale image as a sidecar container.

It actually worked perfectly without any problems at all. I could instantly send requests to my HTTP server from containers in my cluster. Perhaps the craziest part for me is that latency from us-west-2, Oregon, to my computer here in Colorado was only about 80 milliseconds. The internet is amazing.

Wrapping up

I did discover that auth tokens in Tailscale have a max life of 90 days. And there’s no API for renewing them programmatically. I’d have to replace the token manually every 90 days unless I find a way to automate that through my browser or something.

Regardless, now I have image generation and text generation with some limited capability available to my experiments, all for a fraction of the price of having to pay a provider for every request. Since the open source, single computer models aren’t as capable, I might not change any projects, but maybe I will. Who knows!

Thanks for reading. See ya’ll in the next one.

As part of the project I mentioned in my last post, files are uploaded from the user and stored in an S3 bucket. Then I need to process them and do something with that information. The type of file that gets uploaded is probably measured in the hundreds or thousands per user. And the processing might take several seconds or more per file and ideally, the users can upload as many of them at once as they have. I don’t want users to be worried about throttling themselves when they start uploading. As part of my plans, I want a desktop side client that just finds all the files in a folder and uploads them all.

One thing that I could do is only do the processing in bulk in some batch processing workflow. That’s lame though. I don’t want users, aka I don’t want to have to wait a day or a few hours to see the results of the analysis after I upload it. Maybe one day I actually have to do that to handle the scale, but if that’s the case, that’s an amazing success with the project. For now, instead what I’m doing is I’m just queueing up the upload events and putting them into an SQS queue. In my terraform, that basically looks like this:

resource "aws_s3_bucket" "project" {
  bucket_prefix = local.project_bucket_name
}

resource "aws_sqs_queue" "project" {
  name_prefix               = "project"
  message_retention_seconds = 900
}

resource "aws_s3_bucket_notification" "project" {
  bucket = aws_s3_bucket.project.id

  queue {
    events    = ["s3:ObjectCreated:*"]
    queue_arn = aws_sqs_queue.project.arn
  }
}

# And some permissions stuff that I didn't include

I have my S3 bucket, and then create an SQS queue and attach an s3:ObjectCreated notification on it. That means that every time an object is uploaded, a message gets put into an SQS queue with the full key to that new object.

Now, in my fever dream where I get thousands of users all trying to upload their backlog of files all at the same time, I can handle an unlimited number of upload messages. S3 shouldn’t have any problems keeping up with the direct uploads, and the SQS queue should be able to hold all the messages.

But Aaron, if you have unlimited messages in the queue, or just a ton of them, and it takes a few seconds per file to be processed, won’t that mean that your users will have to wait hours or days anyways? Just like you said you wanted to avoid with doing batch processing.

You’re right, listener person, and frankly I’m getting kind of sick of you pointing out all my flaws. Alas, we go on. I continue my adversarial relationship made up of these hallucinated conflicts.

Scaling the consumers

In my post about scaling from 0, I setup the KEDA scaler which has way more plugins other than just queuing up HTTP requests. Right now, the AWS SQS Queue scaler is looking mighty tasty. Unfortunately, it only supports scaling by queue depth. I really wish it would scale based on the average age of messages in the queue. I think that’s a much better metric for understanding the SLA/user experience of people who are waiting for their messages to be processed. We’ll make do though. This is already more work than I’ll probably ever need but it’s kind of the minimum need to make me feel good about the system’s design.

There’s a second order effect for why having this autoscaler is awesome. With SQS, one of the components to pricing is the number of API requests. Even if you make a request for messages and there aren’t any messages, you pay for that. The most common cause for a high number of SQS requests are empty receives. The KEDA scaler lets me scale these consumers down to 0, so if there isn’t any work to do, then there are no workers that are sending long poll requests.

Wrapping up

The last thing I want to talk about is the IAM permissioning of the KEDA controller. According to the configuration, there’s a way to use the identity of the application that is being scaled out to read the SQS metrics that are used for scaling. In practice, this is a little inconvenient because the way it works is that you have to have the trust policy to make the KEDA controller able to do the role assumption to the role of the application. I got lazy and instead of doing the extra step with the role assumptions, I just configured this scaler to use the permissions of the KEDA controller itself and then gave the KEDA controller identity the ability to read metrics about the relevant SQS queue. I don’t like having the permissions on this controller but… it’s my personal environment that no one else uses and it’s a read only set of permissions so meh.

That’s it for this post. Thanks for reading and see ya’ll for the next one.

I’m working on a small side project to automate something for myself. I think it’ll be useful to other people and so I took this as an opportunity to build it such that other people would also be able to use it if they wanted to. It’s a much larger project than my usual afternoon long or weekend long projects, which means I can milk multiple posts out of it if I wanted to. I’ll try to spread them out though if I have other things that I’ve worked on.

I’ve worked at multiple companies now that have used Auth0 for user management. Conceptually, I understand fairly well how JWT based authentication works. I also understand things like machine-to-machine, etc. I’ve built websites with user management on other platforms, but I’ve never actually built anything with Auth0 for myself from scratch. Something new for me, delightful. Ideally, I’d have done more of this in Terraform to keep multiple environments in-sync but I’ve actually been pretty lazy with this project compared to what I normally keep as my standard. I don’t even run anything locally when developing this project. I’m sure one day I should fix that but my push to live pipeline takes about 60-70 seconds. I know that can be a pretty long iteration time, but everything I have written to this point is really simple so far so… meh.

I created the Auth0 application, configured for a React Single Page Application, aka client side only JWTs. Easy enough. Now though… I want to store some user specific information. Like for example, I want user’s to be able to login with multiple different identity providers but still have the same account on my backend. If someone logs in with Twitch credentials or with Discord credentials (it’s a gaming related website), I want them to be considered the same user. So I need to track a user ID on my side of things that will be mapped from their individual identity provider logins.

Auth0 actions

Auth0 has a login flow where you can hook in and modify the JWT or do some other minor actions before anything ever reaches your service.

Every time someone logs in, a request is sent to my backend which then checks to see if this email has ever been seen before. If the email hasn’t, we generate a new “account” on my database, and return a unique ID for that user. If the email has already been seen, then we fetch the previously used unique ID and send it back. Simple enough. Right? Right? You know that every time I say “simple enough”, there’s like a 50% chance that it’s not actually that simple.

Auth0 Actions send that HTTP request but unlike other webhook style workflows like from Stripe or Slack, the Auth0 Action requests don’t come with any kind of verifiable signature. So people could send requests to my endpoint and I’d just blindly assume that they are from Auth0. After reading through the docs, I found that the way to go about this is to add a secret to the action that I’d then verify from my server. I wish I didn’t have to do this but at least it’s not that much extra work.

Verifying Auth0 tokens

Last thing I wanted to mention in this post is verifying the tokens. One of the values of using a JWT is that they’re verifiable completely on their own. You don’t have to make a request to a third party service in order to know that it’s a valid token, which is important for things like access tokens that you might use to grant access to private information.

However, unbeknownst to me, Auth0 may actually not grant a JWT for an access token. If you do not specify an audience when creating your client, you will instead receive an opaque token that doesn’t have any information on its own. That means that you have to send this request to Auth0, specifically their /userinfo endpoint in order to get user information from the access token. I thought about setting an audience or even just sending the id token while all I care about right now is authentication, not authorization, but instead I just got lazy and started sending requests to userinfo.

This was probably a mistake because it turns out that the userinfo endpoint has a fairly low rate limit. I don’t know exactly what it is but I kept hitting it a lot, so I just threw an in-memory cache on my server side. These tokens are valid for 24 hours by default, so having a 5 minute cache isn’t going to be a problem.

Wrapping up

That’s all for now. Thanks for reading and I’ll see ya’ll for the next one.

While writing this, I’m still in between jobs. I wanted to still take a rest while on the break but also I wanted to do something to keep myself from getting too restless. So, I embedded some anime text!

So, what is a text embedding? Simply put, machine learning doesn’t do that well with just raw text itself. Machine learning and AI techniques usually involve multiplying matrices and what not. Math stuff. Text embeddings are representations of text in large arrays, so that we can do math stuff to them. At the end of my last post, I did “announce” (kek, to my ~100 readers), that I was going to be starting a job at Cohere AI. To be very explicit again, when I did this work, I had not started the job yet. This was my own work.

Anyways, Cohere has an embeddings endpoint. And a few weeks ago, Cohere released a big data set of embeddings of all of Wikipedia. While on my break, I’ve been watching a lot of anime to pass the time, and I wondered what kind of experiments I could do revolving around that. And so I scraped the synopsis details of every anime I could get my hands on, and then pushed those through the embeddings endpoint on the Cohere API.

Scraping MAL

I use MyAnimeList.net for tracking the anime that I watch, but also they have synopses written up for all the anime they keep track of. I was stoked to even find out that they have an official API. I thought this was going to be a breeze, until I realized that they don’t have a batch or bulk list endpoint. You have to query each ID for an anime one at a time. Not only that, they don’t expose a way for you to know how many anime there are. Even more, they ask to wait between 500ms - 1000ms per request and they don’t have any formal way to actually rate limit you, so it’s more of an honor system.

The API itself is also very similar to a GraphQL style API, but it’s not. If you make a request to a single anime, you’ll get a few fields by default. You have to request additional fields with a query parameter. At first, I tried requesting every single field and had a for loop to go from 1 to 100,000 with a delay of 500ms between requests. I hit run and then a minute or two later, I get a message from a friend that knew I was doing this project, and she points out to me that MAL has stopped loading, and I check my script and am seeing dozens of timeouts. Sorry MyAnimeList! I don’t know this for sure but I’m going to guess that requesting every field must have some kind of multiplicative database query effect. So instead I reduced the fields that I was requesting just to the synopsis and alternative names and I slowed the requests to doing 1 per second.

for i := 0; i < 100000; i++ {
	time.Sleep(1 * time.Second)

	anime, resp, err := c.Anime.Details(ctx, i,
		mal.Fields{
			"alternative_titles",
			"synopsis",
		},
	)
        // ...the rest of it...
}

The very last ID that I got back was 55,254 but there were only 20,051 actual valid entries. Many of the IDs ended up returning a 404.

Making the embeddings

The Cohere API mentions that for best performance that you should use text snippets that are 512 tokens. I measured and there were something like 20 synopsis that had more than 512 tokens, which you can choose to truncate at the API level but I decided to just go for it. The other nice thing about the API is that you can actually send requests to embed in batches. So that’s what I did. I sent 20,051 synopses but did so 96 at a time. A few minutes and $20 later, I had the embeddings for all of the anime synopses!

What do I do with the embeddings? Originally, I had the idea of trying some kind of recommendation or natural language search engine over the data set, but I “ran out of time” in the sense that I ran out of attention span. Maybe I’ll come back to it. However, I did yoink the simplified nearest neighbor code from the inspiration Cohere blog post and tried a few examples.

What do you want to see?: a pokemon trainer wants to be the very best
Pokemon
Pokémon are peculiar creatures with a vast array of different abilities and appearances; many people, known as Pokémon trainers, capture and train them, often with the intent of battling others. Young Satoshi has not only dreamed of becoming a Pokémon trainer but also a "Pokémon Master," and on the arrival of his 10th birthday, he finally has a chance to make that dream a reality. Unfortunately for him, all three Pokémon available to beginning trainers have already been claimed and only Pikachu, a rebellious Electric-type Pokémon, remains. However, this chance encounter would mark the start of a lifelong friendship and an epic adventure!

Setting off on a journey to become the very best, Satoshi and Pikachu travel across beautiful, sprawling regions with their friends Kasumi, a Water-type trainer, and Takeshi, a Rock-type trainer. But danger lurks around every corner. The infamous Team Rocket is always nearby, seeking to steal powerful Pokémon through nefarious schemes. It'll be up to Satoshi and his friends to thwart their efforts as he also strives to earn the eight Pokémon Gym Badges he'll need to challenge the Pokémon League, and eventually claim the title of Pokémon Master.

[Written by MAL Rewrite]

Pokemon Best Wishes!
As with both the Advanced Generation and Diamond & Pearl series before it, the Best Wishes! series begins with only Satoshi, headed off to the Isshu region, located far away from Kanto, Johto, Houen, and Sinnoh, with his Pikachu. After he meets up with the new trainer and rival Shooty and the region's Professor Araragi, he gains traveling companions in Iris, a girl from a town known for its Dragon Pokémon, and Dent, Pokémon Connoisseur and the Grass Pokémon specialist of the three Sanyou City Gym Leaders.

Pokemon Sun & Moon
After his mother wins a free trip to the islands, Pokémon trainer Satoshi and his partner Pikachu head for Melemele Island of the beautiful Alola region, which is filled with lots of new Pokémon and even variations of familiar faces. Eager to explore the island, Satoshi and Pikachu run wild with excitement, quickly losing their way while chasing after a Pokémon. The pair eventually stumbles upon the Pokémon School, an institution where students come to learn more about these fascinating creatures.

At the school, when he and one of the students—the no-nonsense Kaki—have a run-in with the nefarious thugs of Team Skull, Satoshi discovers the overwhelming might of the Z-Moves, powerful attacks originating from the Alola region that require the trainer and Pokémon to be in sync. Later that night, he and Pikachu have an encounter with the guardian deity Pokémon of Melemele Island, the mysterious Kapu Kokeko. The Pokémon of legend bestows upon them a Z-Ring, a necessary tool in using the Z-Moves. Dazzled by their earlier battle and now in possession of a Z-Ring, Satoshi and Pikachu decide to stay behind in the Alola Region to learn and master the strength of these powerful new attacks.

Enrolling in the Pokémon School, Satoshi is joined by classmates such as Lillie, who loves Pokémon but cannot bring herself to touch them, Kaki, and many others. Between attending classes, fending off the pesky Team Rocket—who themselves have arrived in Alola to pave the way for their organization's future plans—and taking on the Island Challenge that is necessary to master the Z-Moves, Satoshi and Pikachu are in for an exciting new adventure.

[Written by MAL Rewrite]

I did also think about trying the new Cohere rerank API, but my lizard brain said this project was nearing an end. I had but one more thing to do.

Uploading to HuggingFace

I’ve downloaded things through the HuggingFace SDKs before but I’ve never uploaded anything. It was easy though. You need to setup git with large file storage and then you basically just treat everything like a git repo. ezclap.

I uploaded just a simple jsonlines file, instead of doing anything more complicated or compressed and HuggingFace was able to parse it and show a preview on the README.

Wrapping up

Cool. Thanks for reading. Check out the dataset and code to get started: https://huggingface.co/datasets/abatilo/myanimelist-embeddings

See ya’ll for the next one.

Now, normally, I use Karpenter.sh as my cluster’s node management tool. I have a provisioner configured with a ttlSecondsUntilExpired that will delete instances if they’ve been around for more than 24 hours. The instances are already spot instances so they’re not as likely to even stick around for that long, but for low utilization instance types like the t3.mediums that I use, I’ve had instances stick around for months if I don’t do something about it.

Okay, so, Aaron, you’ve already got something that deletes instances that are old, so why are you making me read any of this? Well, it was decided that Karpenter should not be schedulable on any nodes that it provisions, by default. I think that kind of makes sense though. If for some reason there’s a problem with your provisioner, you don’t want Karpenter itself to be unable to run because there are no nodes. That means that while Karpenter can handle all additional worker nodes that need to serve your application requirements, you need to have some workers join your EKS cluster via other means.

The terraform-aws-eks module that I use for provisioning my cluster comes with a handy dandy built in variable for configuring the official EKS managed node groups. EKS managed node groups are a regular old autoscaling group with some extra bells and whistles. Here’s a description of the differences written by one of my newest favorite tools, Perplexity.ai:

An AWS Autoscaling Group and an AWS EKS managed node group are both used for automatic scaling and management of EC2 instances in an EKS cluster. However, there are some differences between them. An Autoscaling Group is a collection of EC2 instances that are treated as a logical grouping for the purposes of automatic scaling and management¹. It lets you use Amazon EC2 Auto Scaling features such as health check replacements and scaling policies. On the other hand, an EKS managed node group automates the provisioning and lifecycle management of nodes for your Kubernetes clusters². It provides an abstraction to Amazon EC2 instances and Auto Scaling groups, enabling a set of simple one-step operations in EKS to provision and manage groups of cluster nodes. It is compatible with the Cluster Autoscaler and is backed by EC2 instances in your account which are managed by an Auto Scaling group

But firstly, what’s wrong with the current setup? Why can’t some of these instances be a little bit more stable? WHAT PROBLEM ARE YOU TRYING TO SOLVE, AARON? Well you see, what’s been happening is that I kept this base node group at a desired size of 2 instances. One for each replica of Karpenter itself. Then any additional nodes that existed for my actual applications were provisioned by Karpenter. My cluster almost always runs with 3 nodes right now with how much compute I give each website, things like ArgoCD, etc. However, every 24 hours, the 1 node that’s managed by Karpenter would get terminated and replaced, and all of the pods that were scheduled to that node would end up moving to the existing nodes.

This would happen repeatedly and then after a few weeks of this happening every day, I’d get unlucky and end up with too many pods on a single node, and trigger problems with the container network interface trying to assign more IP addresses to a given node than is allowed. So I’d have to manually terminate those nodes to get new ones to rebalance the cluster. And this last time that I did so, I had a cascading failure, where my scale from 0 autoscaler had a bunch of requests queue’d up, which triggered a bunch of pods to be created, which made everything worse and zero Karpenter pods could get scheduled and so no new nodes came up. This is basically the worst case combination of things happening.

Let’s make these EC2 instances disappear after 24 hours also.

AWS Autoscaling Groups themselves have a configuration for MaxInstanceLifetime which is exactly what I want for these extra nodes. There’s just one problem, the abstraction for the EKS managed node groups doesn’t expose this property at all. So I could go back to using raw autoscaling groups for my nodes, but then I lose all the bells and whistles of the EKS managed node groups… so what do I do?

Like all good infra people know, you glue together a solution with some bash.

#!/bin/bash
asgs="$(aws autoscaling describe-auto-scaling-groups --filters Name=tag:kubernetes.io/cluster/red,Values=owned --query 'AutoScalingGroups[*].AutoScalingGroupName' | jq -r '.[]')"

# Loop through each asg in asgs and call aws autoscaling start-instance-refresh
for asg in $asgs; do
  aws autoscaling start-instance-refresh --auto-scaling-group-name $asg --preferences '{"InstanceWarmup":90,"MinHealthyPercentage":66}'
done

Viola! Let’s just grab all the autoscaling groups that are owned by my cluster (named “red”), and then trigger an instance refresh. Put this script into a container, then run it as a CronJob from within the EKS cluster itself and bing bang boom.

As an homage to Leonardo himself, I was hoping to have the instance refresher run once every 25 hours, but alas, it doesn’t appear that cron expression evaluators can actually work that way, even if it’s syntactically valid to say */25.

As an aside, by the time you’re reading this newsletter post, I’ve already publicly announced that I’ve left my position as an infra and tools engineer with Color. I’m ecstatic to share that I’ll be joining Cohere to do some infra work there. I’m very excited to be working full time in the large language models space, and you can expect that working in such a space will inspire some pretty interesting experiments and learnings for me to write about in the future!

Motivation

The motivation for all this work comes from a few places. First of all, I’ve always been super interested in machine learning. People who know me know that I’ve been experimenting with machine learning for many many years. But I have another love. And that’s Super Smash Bros Melee. Mang0 is the GOAT. More on this in a moment. A few weeks ago, I watched the AlphaGo documentary, where DeepMind did what was thought to be impossible, and they trained an AI that beat the long time world Go champion. DeepMind has done a lot of amazing projects, and a lot of their success has been from applications of reinforcement learning.

I’ve also been a casual chess fan. I have a 400 rating, am terrible at the game, but something about watching Levy “GothamChess” Rothman is very entertaining. A sentiment that chess players used to have, and that Go players in the documentary started to have, is to wonder if AI’s beating humans will mean that people will stop wanting to play the game. In fact, it’s been the opposite. AI has been found to be immensely useful in chess players being able to study the game. Just because a computer can calculate more moves ahead, humans are still fascinated in finding out how capable they can be given our limits.

Back to Melee. It’s the greatest game to have ever been created. It’s expressive, it’s fast, it’s beautiful. I love watching this game. I’m a bronze 2 shitter, but I still love watching this game. Let’s put this all together. I wondered to myself, is there a way that I can build a melee AI that will ultimately help push the meta forward? With the fearless exploratory capabilities that a reinforcement learning algorithm has, can we find new tech? If I trained a model for low tier characters, can we make them more relevant? I have no idea, but I’m interested in finding out.

Thus, I get started. This will not be the only thing I talk about in the coming weeks on this newsletter, but it might be a lot of the next few posts. Everything from previous RL work on Melee to scaling out model training, etc. I expect this project to be a running theme for a while but I’m too scatter brained to keep to a single long term project.

How do I get started with RL?

A few months ago, I read this article that says that OpenAI uses Ray.io. When I checked out the Ray documentation, I noticed that it has reinforcement learning capabilities. At the time, I took that information and stored it back in the brain archive room. Fast forward to when I’m interested in trying some RL work and I decided that Ray is where I want to get started. Ya’ll know that I’m an operations person. Ya’ll know that I run my own Kubernetes cluster. My assumption is that by the time I’m ready and capable of doing simulations in melee, that I’ll want to be able to distribute that work, and since Ray seems to be the hot shit here, I might as well start there.

So I start searching for tutorials and trying to follow them. And I notice something. None of these tutorials work. Either the tutorials aren’t for Ray, or I follow the steps and the provided code doesn’t run, or they don’t specify that you need a certain version of Python, etc. It’s maddening. I spent probably a full day and a half strictly trying just to get something running. I’m not an ML engineer, and the learning curve for getting started here feels way too hard, so I wanted to write about everything I learned.

Laying out the tools

Part of what made it really hard for me to get started was that there’s a set of standard terms and tools that all of these tutorials kind of just expect you to be familiar with. I was not, and I’m still not entirely sure I understand many of the specifics, but I wish someone had laid some of these things out for me when I was getting started.

I think of reinforcement learning as having 4 key components: Environment, Agent, Policy, Reward.

Environment - Often a simulation when training, but otherwise, the environment the world that is being observed and acting in. This can be a game, the real world, a description of a problem, etc.

Agent - This is the representation of the program. This might be the self driving car, this might be your character in a video game. An agent takes actions on an environment. In a fighting game, an agent has a move set to follow, and that’s all they can do on their own.

Policy - The policy is the algorithm or strategy for what an agent will do given a certain state of the environment. When you’re training a reinforcement learning model, you’re most likely to be referring to learning a new policy. When you play a fighting game, you are actually the policy.

Reward - There needs to be some kind of evaluation or feedback. The reward and cost model is going to dictate how the policy behaves.

When you put all of these 4 together, you have a policy that looks at the current environment, tells the agent to take an action, and the environment will change in some way which can give you a reward or punish you.

You’ll also have episodes / rollouts. These terms are mostly interchangeable from my understanding. Basically, this is one full simulation through an environment. So this could be a single chess match as one example.

One of the first tools that you’ll see everywhere when you try to get started with reinforcement learning is OpenAI’s gym. Gym is a platform and set of abstractions for having a RL environment. It’s definitely become an RL staple and the standard way for everyone to integrate everything else. There are interfaces between gym and things like the Unity game engine, game console emulators, and more. Here’s the catch, OpenAI gym has actually ceased development. Gym has been locked in place and now all development is done under the Farama Foundation’s Gymnasium repository. Gymnasium does its best to maintain backwards compatibility with the gym API, but if you’ve ever worked on a software project long enough, you know that dependencies get really complicated. Ray is a modern ML framework and later versions integrate with gymnasium well, but tutorials were written expecting gym.

Okay, so that’s a quick overview of gym and gymnasium. What else are we using? Ray! So what’s up with Ray? Ray is a combination distributed workflow system with overlap with projects like Apache Spark, or Dask, but with some additional features and focus on machine learning workloads. The Ray ecosystem itself integrates deeply with many other libraries, but something that makes their documentation a little confusing to a total newbie, is that Ray loves its own ecosystem. What does that mean? Ray documentation and examples take every opportunity to use other parts of the Ray ecosystem. Examples might show you how to do something with Ray’s RLLib, but then will also give you examples using RayTune and RayServe, etc. Overall, I think this is awesome, but very confusing when you’re still learning all of the boundaries and what they do.

RayTune is a library for doing parameter grid searches and optimizations. In other words, it helps you find optimal hyper parameters for your model. Conceptually, think about it as a library to help you run more experiments. RayServe will help you turn your models into a REST API so that you can send requests to your model from other services.

Lastly, I’ll make a quick call out to other ML frameworks. Ray integrates with both tensorflow1, tensorflow2, and torch. Again, if you’re an ML engineer, you’re probably comfortable with these, but when you’re just an admirer/fanboy, it’s hard to keep up with all of these data science/ml libraries. Simply put, these libraries make it easier to architect neural networks, and since Ray supports both, you’ll find examples that use both but that can cause even more confusion with dependency management. Numerical computing libraries are incredibly finicky with their own dependencies, and if you try to install both tensorflow and torch like the Ray tutorial tells you to, you’ll likely end up with a conflict of versions of shared dependencies that both libraries use.

It’s awesome that Ray wants to demonstrate compatibility with both, but man did that make things confusing.

Making a code sample reproducible

If you type pip install ray which is so common in documentation or tutorials, you’ll possibly install the latest version, or maybe you followed a tutorial 6 months ago and pip will just report that it’s already installed with whatever version you had. You could force a new version with the -U flag but that doesn’t really solve the problem. You have multiple dependencies that are going to be installed and just installing “ray” isn’t going to do anything deterministic or predictable for you.

Let’s scope in what model we want to train to something very specific. I’m going to help you run this code which we want to save as main.py:

import gymnasium as gym
from ray.rllib.algorithms.dqn import DQNConfig

algo = DQNConfig().environment("LunarLander-v2").build()

for i in range(10):
    result = algo.train()
    print("Iteration:", i)
    print("Episode reward max:", result["episode_reward_max"])
    print("Episode reward min:", result["episode_reward_min"])
    print("Episode reward mean:", result["episode_reward_mean"])
    print()

env = gym.make("LunarLander-v2", render_mode="human")
terminated = truncated = False
observations, info = env.reset()

while True:
    env.render()
    action = algo.compute_single_action(observations)
    observations, reward, terminated, truncated, info = env.step(action)

    if terminated or truncated:
        observations, info = env.reset()

That’s it. I’m going to write this big-ass post to help you run this. What does this do? There’s a classic reinforcement learning “hello world” type problem called the LunarLander.

We run 10 training iterations where each training iteration runs the default number of episodes/rollouts. Then we have an infinite loop where we render and display the trained policy running on the LunarLander game. We’ll be digging into each section of code more deeply in the coming sections.

I just want to run it, Aaron!

Okay, okay, so what do you need to install it? I’ve generated the FULL dependency tree to run this example consistently using pip-tools. I am running everything with Python 3.8. I specifically use asdf-python for managing multiple versions of Python, but feel free to use pyenv. Once you’re running Python 3.8, save the following content to a file named requirements.txt:

#
# This file is autogenerated by pip-compile with Python 3.8
# by the following command:
#
#    pip-compile --output-file=requirements.txt requirements.in
#
absl-py==1.4.0
    # via
    #   tensorboard
    #   tensorflow
aiosignal==1.3.1
    # via ray
astunparse==1.6.3
    # via tensorflow
attrs==22.2.0
    # via
    #   jsonschema
    #   ray
box2d-py==2.3.5
    # via gymnasium
cachetools==5.3.0
    # via google-auth
certifi==2022.12.7
    # via requests
charset-normalizer==3.1.0
    # via requests
click==8.1.3
    # via
    #   ray
    #   typer
cloudpickle==2.2.1
    # via
    #   gym
    #   gymnasium
contourpy==1.0.7
    # via matplotlib
cycler==0.11.0
    # via matplotlib
distlib==0.3.6
    # via virtualenv
dm-tree==0.1.8
    # via ray
farama-notifications==0.0.4
    # via gymnasium
filelock==3.10.6
    # via
    #   ray
    #   virtualenv
flatbuffers==23.3.3
    # via tensorflow
fonttools==4.39.2
    # via matplotlib
frozenlist==1.3.3
    # via
    #   aiosignal
    #   ray
gast==0.4.0
    # via tensorflow
google-auth==2.16.3
    # via
    #   google-auth-oauthlib
    #   tensorboard
google-auth-oauthlib==0.4.6
    # via tensorboard
google-pasta==0.2.0
    # via tensorflow
grpcio==1.51.3
    # via
    #   ray
    #   tensorboard
    #   tensorflow
gym==0.23.1
    # via ray
gym-notices==0.0.8
    # via gym
gymnasium[box2d]==0.28.1
    # via -r requirements.in
h5py==3.8.0
    # via tensorflow
idna==3.4
    # via requests
imageio==2.26.1
    # via scikit-image
importlib-metadata==6.1.0
    # via
    #   gym
    #   gymnasium
    #   markdown
importlib-resources==5.12.0
    # via
    #   jsonschema
    #   matplotlib
jax==0.4.6
    # via tensorflow
jax-jumpy==1.0.0
    # via gymnasium
jsonschema==4.17.3
    # via ray
keras==2.12.0
    # via tensorflow
kiwisolver==1.4.4
    # via matplotlib
lazy-loader==0.2
    # via scikit-image
libclang==16.0.0
    # via tensorflow
lz4==4.3.2
    # via ray
markdown==3.4.3
    # via tensorboard
markdown-it-py==2.2.0
    # via rich
markupsafe==2.1.2
    # via werkzeug
matplotlib==3.7.1
    # via ray
mdurl==0.1.2
    # via markdown-it-py
msgpack==1.0.5
    # via ray
networkx==3.0
    # via scikit-image
numpy==1.23.5
    # via
    #   contourpy
    #   gym
    #   gymnasium
    #   h5py
    #   imageio
    #   jax
    #   jax-jumpy
    #   matplotlib
    #   opt-einsum
    #   pandas
    #   pywavelets
    #   ray
    #   scikit-image
    #   scipy
    #   tensorboard
    #   tensorboardx
    #   tensorflow
    #   tifffile
oauthlib==3.2.2
    # via requests-oauthlib
opt-einsum==3.3.0
    # via
    #   jax
    #   tensorflow
packaging==23.0
    # via
    #   matplotlib
    #   scikit-image
    #   tensorboardx
    #   tensorflow
pandas==1.5.3
    # via ray
pillow==9.4.0
    # via
    #   imageio
    #   matplotlib
    #   scikit-image
pkgutil-resolve-name==1.3.10
    # via jsonschema
platformdirs==3.2.0
    # via virtualenv
protobuf==3.20.3
    # via
    #   ray
    #   tensorboard
    #   tensorboardx
    #   tensorflow
pyasn1==0.4.8
    # via
    #   pyasn1-modules
    #   rsa
pyasn1-modules==0.2.8
    # via google-auth
pygame==2.1.3
    # via gymnasium
pygments==2.14.0
    # via rich
pyparsing==3.0.9
    # via matplotlib
pyrsistent==0.19.3
    # via jsonschema
python-dateutil==2.8.2
    # via
    #   matplotlib
    #   pandas
pytz==2023.2
    # via pandas
pywavelets==1.4.1
    # via scikit-image
pyyaml==6.0
    # via ray
ray[rllib]==2.2.0
    # via -r requirements.in
requests==2.28.2
    # via
    #   ray
    #   requests-oauthlib
    #   tensorboard
requests-oauthlib==1.3.1
    # via google-auth-oauthlib
rich==13.3.2
    # via ray
rsa==4.9
    # via google-auth
scikit-image==0.20.0
    # via ray
scipy==1.9.1
    # via
    #   jax
    #   ray
    #   scikit-image
six==1.16.0
    # via
    #   astunparse
    #   google-auth
    #   google-pasta
    #   python-dateutil
    #   tensorflow
swig==4.1.1
    # via gymnasium
tabulate==0.9.0
    # via ray
tensorboard==2.12.0
    # via tensorflow
tensorboard-data-server==0.7.0
    # via tensorboard
tensorboard-plugin-wit==1.8.1
    # via tensorboard
tensorboardx==2.6
    # via ray
tensorflow==2.12.0
    # via -r requirements.in
tensorflow-estimator==2.12.0
    # via tensorflow
tensorflow-io-gcs-filesystem==0.31.0
    # via tensorflow
termcolor==2.2.0
    # via tensorflow
tifffile==2023.3.21
    # via scikit-image
typer==0.7.0
    # via ray
typing-extensions==4.5.0
    # via
    #   gymnasium
    #   rich
    #   tensorflow
urllib3==1.26.15
    # via requests
virtualenv==20.21.0
    # via ray
werkzeug==2.2.3
    # via tensorboard
wheel==0.40.0
    # via
    #   astunparse
    #   tensorboard
wrapt==1.14.1
    # via tensorflow
zipp==3.15.0
    # via
    #   importlib-metadata
    #   importlib-resources

# The following packages are considered to be unsafe in a requirements file:
# setuptools

Then to make sure that you’re not going to be having any other dependencies conflict with these, make sure to run in a new virtual environment. I’m not going to go into the details of virtual environments, there’s plenty of documentation about them. But basically after you save the above contents, just run.

python -m virtualenv venv
. venv/bin/activate
pip install -r requirements.txt
python main.py

This will start the training loop. What were the high level dependencies that we installed? I generated that pinned version tree after installing the following dependencies:

ray[rllib]
gymnasium[box2d]
tensorflow

We install ray and the rllib extras, we install gymnasium for our environment and the box2d extras, and then tensorflow. So we install the general ML framework in tensorflow, we install the reinforcement learning framework in ray with rllib, and then we install the game environment with gymnasium and box2d. Box2d is going to be really valuable here because it installs pygame as a downstream dependency which will give us the actual game rendered on screen to visualize our trained algorithm which I think is at least half the fun. When you run the main.py, we’re currently only running 10 train loops which isn’t going to be enough to train a consistently functioning model, but you can change that number as you experiment.

Breaking down our code

The documentation for LunarLander on the Gymnasium website is actually amazing. I recommend just reading through that to understand all of what the environment actually entails. But in the context of our training loop, I want to emphasize one bit of text from the documentation.

Q learning and Deep Q learning

So in our code, we start with:

algo = DQNConfig().environment("LunarLander-v2").build()

So what is this doing? This creating an empty policy for a Deep Q Network, targeted towards the LunarLander environment. We just talked about the LunarLander, but we need to talk about Deep Q Networks and what this line of code is actually doing in the ray rllib library.

Q learning is one of the many many many reinforcement learning algorithms. I’m fucking AWFUL at math, so I’m not even going to attempt to derive that or explain it. But Q learning is simply the idea that you have some function named Q, and it takes two parameters. It takes the current state of the observable environment, an action to take, and it will give you an estimation for what the reward will be. Pretty simple. So in a plain Q learning setup, you can imagine that at every step of the game, you check the return value of your Q function for every action, and you just pick the action with the best prediction. If you have a very small environment space to work through, you could imagine literally hard coding the inputs and outputs of your Q function. Imagine writing a function that adds two numbers. If you know that your inputs will only ever be 0 or 1, you could write a function like so:

def add(a, b):
  if a == 0 and b == 0:
    return 0
  if a == 0 and b == 1:
    return 1
  if a == 1 and b == 0:
    return 1
  if a == 1 and b == 1:
    return 2

A Deep Q Network applies a neural network for trying to learn more generalized reward calculation given more complicated states. Any practical usage of reinforcement learning is going to have way more nuances to how you calculate your output, but the basics of Q learning come from a very easy to understand concept. It makes for a great starting algorithm for a noob like me.

That being said, DeepMind was able to train a single model that could play most if not every Atari games using a Deep Q Network. So it’s also an incredibly powerful algorithm when used by people who know what they’re doing.

Training your model and counting your rewards

The line of code that I have is as simple as it can get. You can actually pass in all kinds of additional configuration here for Ray to understand how to distribute the work of this algorithm. The DQNConfig is where you’d configure things like GPU allocations, number of episodes in a single training iteration, etc. But I’m going to skip that for now because this post is already dummy thiqq.

for i in range(10):
    result = algo.train()
    print("Iteration:", i)
    print("Episode reward max:", result["episode_reward_max"])
    print("Episode reward min:", result["episode_reward_min"])
    print("Episode reward mean:", result["episode_reward_mean"])
    print()

The next chunk of our code here is to do our training. A single call to train() will actually trigger thousands of simulations by default, then will report some statistics. Notably, we’re going to print out the episode reward. We noted earlier that a success for LunarLander takes a reward of 200. If you run this for 10 iterations, you’ll probably only see negative reward values here. In my experience, if you run with 30-50 iterations, you’ll at least see a reward max of 150-200, but that reward mean is going to stay negative. After a few hundred train loops, you’ll see a mean of over 200 but you might still have the occasional rollout with a large negative number.

Watching your algorithm go to work

env = gym.make("LunarLander-v2", render_mode="human")
terminated = truncated = False
observations, info = env.reset()

while True:
    env.render()
    action = algo.compute_single_action(observations)
    observations, reward, terminated, truncated, info = env.step(action)

    if terminated or truncated:
        observations, info = env.reset()

The last section of code is here to let us visualize how our algorithm is doing. The first line here is instantiating an entirely new gym environment for the LunarLander game, and setting the render_mode to “human”. Render modes were another thing that were really confusing to me because there’s convention about it but you’ll finds all kinds of documentation for custom render modes. Basically, the render_mode controls how the environment will… well, be rendered. The convention is that setting the render_mode to “human” will do whatever the environment needs for being viewable by human eye balls. In the case of LunarLander, that’s to render a pygame window to draw the game. By default, the environments won’t actually render anything because that takes more time and more compute. If you’re rendering a non-trivial video game environment, the game is almost guaranteed to run at 60 frames per second, which would mean 60 Q function steps per second. A game might take minutes to complete a full rollout, so this would slow down the training process by a lot. If the game can be run “headless” or purely computationally for simulation purposes, then using that headless render mode would make more sense for being able to do thousands or millions of simulations. Render modes might also attach to real physical devices when people are trying to do some experimenting, which is probably why the argument is just any arbitrary string. I would find so many tutorials that would show me a video of the results of the tutorial, but the tutorial itself NEVER explained render mode which made me think that I copy pasted something wrong or that my system was setup with the wrong dependencies, because I’d never actually see the game.

We create the environment, then we setup some defaults for variables that we’re going to use. Then comes the best part, running the game in a loop. First we call env.render() which is what updates the pygame window. Some RL environments might even just print to your terminal when you call render, but that render_mode being set to “human” has our back here. Next, we pass in the observable states to our trained policy, and it’ll return one of the defined actions according to the environment. Next, apply the action from the policy to the environment which will impact our space craft agent by having it trigger one of its thrusters. Lastly, we check to see if we’ve terminated or truncated our environment. To terminate or truncate the environment usually means something like we crashed, died on the level, or have been running for too long with no progress and got stuck. It’s up to the environment to define these states. Termination can actually also mean that we’ve beaten or won the level. If we hit either case, reset the environment back to the beginning of the level and just keep it going. If you’ve ran enough iterations of the training loop, hopefully that means you get an endless loop of the space craft landing safely between the flags. If not, you might be watching the space craft fly off the side of the screen or plummet to its death over and over again. Remember that these visualizations are not doing anything to update the model. This is purely to visualize the results of the training that has already happened.

Conclusion

Yeah, this post was a chunky boi. But I’m learning a lot with this project and writing this up is exactly the point of this newsletter. Checking my own understanding and trying to make this understandable is fun. It’s flexing a muscle that I should flex more often. Anyways, moving forward from this, I’m going to attempt to understand some more reinforcement learning algorithms. Q learning and DQN is great for starting, but I’m not sure if it’ll be enough to learn melee. Maybe it will be, who knows. At some point, I also need to learn more about what it’ll take to get melee available as a Ray compatible (aka gymnasium compatible) environment so that I can start doing simulations. There’s some previous work about using reinforcement learning with melee but thanks to the melee community, the tech around running melee has never been more advanced. Some of the previous work might apply, some might now. We’ll find out and when I do, you’ll hear about it.

See ya’ll in the next one.

In January, I asked GPT-3.5 49,717 trivia questions from the uberspot/OpenTriviaQA data set. I learned a few things and there have been a few new models since then too.

Let’s get a few corrections in place. In that last post, I claimed that ChatGPT is actually the same as the GPT-3.5 model which is what I had API access to. It turns out that this isn’t entirely true. ChatGPT was running on a model that is now known as GPT-3.5-Turbo. Not only was that model fine-tuned for chat bot type interactions, but the model itself was optimized and presumably quantized. The API costs for the turbo model were about 1/10th the costs of GPT-3.5.

And yes, I was in the middle of running this same test on that model. When I made my original post, someone was kind enough to point out to me that the results of asking the models questions actually has improved results when you ask the model to generate an explanation BEFORE giving an answer. This has been called chain-of-thought. So I was inspired to try both GPT-3.5 and GPT-3.5-Turbo with this technique to see how the results would change. I didn’t actually finish doing any more tests with GPT-3.5 because the results after the first 10-15 thousand questions were almost identical, even with the chain of thought improved prompt. My GPT-3.5-Turbo, aka the confirmed ChatGPT model, did in fact see some improvement, but not much. I won’t share the breakdown this time around but the overall percentage of correct questions went from 66.7% up to 70.1%. Considering the model was 1/10th the cost to operate, that was awesome to have any improvements in correctness at all.

When the GPT-4 release announcement was made, I was quick to request access to the API. I was granted access just a few days later and I immediately went ahead and started sending all of the same trivia questions. This time I did a little bit more work on getting a prompt that showed promise. And this time I actually included a single example of how I wanted the model to explain its thinking.

Both GPT-3.5-Turbo and GPT-4 leverage the chat API format that was released with GPT-3.5-Turbo. This format doesn’t just have you send a prompt anymore, but actually lets you format your messages to look more like a chat history with back and forth between two entities. My actual prompt that I used for this GPT-4 test looks as follows:

Messages: []openai.ChatCompletionMessage{
	{
		Role:    "system",
		Content: `You are a knowledge assistant. I will ask you a multiple choice question and you will answer it.`,
	},
	{
		Role: "user",
		Content: `You will be given a question and then you will be presented with possible
answers to choose from. If you're not sure of the answer, make your
best guess and pick one of the answers.

Follow these instructions:
- Think out loud, step by step
- Insert two blank lines to separate your answer from your explanation
- Write one of your answers and write it exactly character for character as it appears in the list of possible answers
`,
	},
	{
		Role: "user",
		Content: `Question:
What language can Harry Potter speak?

Possible answers:
Goblin
English
Mermish
Parseltounge
`,
	},
	{
		Role: "assistant",
		Content: `Thinking out loud:
Harry never has a need to speak goblin, so I can eliminate that answer. English
is the known language that he speaks which leaves mermish and parseltounge.
Mermish would be the language that merpeople speak, so I can eliminate that
answer. That leaves parseltounge as the correct answer.

Answer:


Parseltounge`,
	},
	{
		Role:    "user",
		Content: questionTemplate,
	},
},

How were the results?

Category	Correct	Total	GPT-3.5 	GPT-4		Delta
brain-teasers	154	207	0.497585	0.743961	49.51%
video-games	415	599	0.517529	0.692821	33.87%
television	4033	5230	0.556597	0.771128	38.54%
entertainment	224	280	0.582143	0.8		37.42%
animals		1083	1366	0.596632	0.792826	32.88%
celebrities	2577	3196	0.597309	0.80632		34.99%
sports		2331	2840	0.608451	0.820775	34.90%
movies		3484	4314	0.613584	0.807603	31.62%
for-kids	648	759	0.638999	0.853755	33.61%
music		4677	5579	0.671447	0.838322	24.85%
literature	1126	1288	0.689441	0.874224	26.80%
hobbies		1059	1242	0.698068	0.852657	22.15%
general		2854	3290	0.700912	0.867477	23.76%
newest		2590	3016	0.701923	0.858753	22.34%
people		2457	2743	0.71965		0.895735	24.47%
technology	2160	2486	0.7321		0.868866	18.68%
world		4322	4875	0.732513	0.886564	21.03%
religion-faith	579	638	0.73511		0.907524	23.45%
history		1481	1645	0.746505	0.900304	20.60%
rated		1987	2185	0.750572	0.909382	21.16%
humanities	965	1097	0.75752		0.879672	16.13%
geography	771	842	0.774347	0.915677	18.25%
overall		41977	49717	0.667377	0.844319	26.51%

Improvements in every category, and pretty big ones at that. In the above chart, we’re still ordering the categories by how successful GPT-3.5 was, but you can see some pretty amazing numbers here. The brain-teasers category improved by almost 50%. The geography category went all the way up to 91.6%. GPT-4 is by every measure in this test, “smarter”. How did the explanations do?

Let’s look at the same questions as last time, but now with the explanations coming BEFORE an answer is picked.

Question:
The name of what is coded in the following: 86,smopuim?

Possible answers:
it does not mean anything
the name of a software product
a NASA aircraft
a secret CIA project


Thinking out loud:
First, I will try to find any pattern or code within the given sequence "86,smopuim". If the number 86 is used as a key for Caesar cipher, I can try to decrypt the part "smopuim". However, even after attempting that, I do not find any meaningful name or phrase.

Answer:

it does not mean anything

The last time we asked this question, GPT-3.5 got the answer correct, but the explanation that we got last time was basically a guess that the numbers 86 would only be used in a software product.

The explanation here actually feels more consistent given that the model wasn’t capable of reading text upside down. Reading text upside down would probably need to happen more in the training data before it would even be a consideration by the model.

Question:
What number multiplied by 10 equals the square of the same number, times 5?

Possible answers:
4
5
2
10


Thinking out loud: 
Let x be the number. Then we're looking for x such that 10x = x^2 * 5. Simplifying the equation, we get 2x = x^2. Now let's check the given numbers.

- If x = 4, we have 2*4 = 8 and 4^2 = 16. This doesn't match.
- If x = 5, we have 2*5 = 10 and 5^2 = 25. This doesn't match either.
- If x = 2, we have 2*2 = 4 and 2^2 = 4. This is a match.
- If x = 10, we have 2*10 = 20 and 10^2 = 100. This doesn't match.

Answer:

2

Last time, this question was answered incorrectly, but this time, GPT-4 actually appears to have been doing the math to solve the equation. I think that this example is actually insane. The model took the word description of the equation to solve and converted it, SIMPLIFIED IT, and then the completed text went through and plugged in the equation.

Question:
A man is carrying three croquet balls across a bridge. The bridge has a sign that says bridge can only hold 200 pounds. The man weighs 195 pounds and the croquet balls weigh 2 pounds each. The man ends up crossing the bridge in one trip and no one else helps him. This is possible.

Possible answers:
False
True


Thinking out loud:
The man weighs 195 pounds, and each croquet ball weighs 2 pounds. He is carrying three croquet balls, which would be a total of 6 pounds added.
We calculate the total weight like this : 195 pounds (man) + 6 pounds (croquet balls) = 201 pounds. Since the bridge can only handle 200 pounds, and the total weight is 201 pounds, it should not be possible.

Answer:

False

In the last test, I noted that this is weird because this riddle has been disproven by people with wrinkly brains that know how physics works. The data set says that the answer here is True. Googling this riddle has the riddle solution listed as something like: “just throw the balls across the bridge” or “just juggle the balls and then you won’t have all 6 pounds of force on the bridge”. GPT-3.5 guessed True but made the assertion that 201 pounds is less than 200 pounds, which is obviously incorrect. GPT-4 correctly asserted all numbers, and responded that the answer is False, which is real world correct, but data set incorrect. I’m going to leave the data set as is, but I’m glad to see that GPT-4 knows that 201 pounds is heavier than 200 pounds.

Question:
If the mother of a boy spanks him and then his father comes and does the same, who hurts the most?

Possible answers:
The boys brother
The mother
The boy
The father


Thinking out loud:
In this situation, the boy is receiving the spanking from both his mother and father. The boy's brother does not have any involvement in the situation, so he wouldn't be hurt. The mother and father are administering the punishment, so they are not the ones receiving the pain. Therefore, the most reasonable answer would be the one who is actually experiencing the physical pain.

Answer:

The boy

GPT-3.5 and GPT-4 answer this question with a very similar explanation. Understanding the context of the actions being taken in the question is still amazing to me.

Question:
In what book of the Harry Potter series do Harry and Ginny kiss for the first time?

Possible answers:
Harry Potter and the Order of the Phoenix
Harry Potter and the Prisoner of Azkaban
Harry Potter and the Deathly Hallows
Harry Potter and the Half Blood Prince


Thinking out loud:
Harry and Ginny don't share their first kiss in the first few books, so I can eliminate the Harry Potter and the Prisoner of Azkaban. Harry Potter and the Deathly Hallows has too many other big events happening, which makes it less likely. It's either Harry Potter and the Order of the Phoenix or Harry Potter and the Half Blood Prince since their relationship develops more in those books. Harry starts to show his feelings for Ginny in the Order of the Phoenix, but I remember that their first kiss happens in the Half Blood Prince when they're celebrating a Quidditch victory.

Answer:

Harry Potter and the Half Blood Prince

Another one that GPT-3.5 and GPT-4 nailed. Not much else to talk about for this one.

Question:
Harry Potter marries Hermione in the last book of the series.

Possible answers:
True
False


Thinking out loud:
Harry Potter and Hermione Granger are close friends in the series, but they do not have a romantic relationship. Harry ends up marrying Ginny Weasley, and Hermione marries Ron Weasley. So, claiming that Harry marries Hermione in the last book is incorrect.

Answer:

False

Last time, GPT-3.5 got this incorrect and I posited that maybe fan fiction or general internet sentiment is what influenced the model to think that Harry Potter and Hermione were married. GPT-4 knew differently this time.

Wrapping up

The patterns that we saw from the last test are the same. Categories with long and well known information and facts still perform the best, but GPT-4’s ability to “reason” through a problem appears to have greatly improved according to the results of the brain teasers category. Rumor has it that GPT-4 is actually a trillion parameter model, as opposed to GPT-3/3.5 which has been confirmed publicly to be a 175 million parameter model. The turbo variant hasn’t had its number of parameters confirmed but it’s safe to assume that its less than GPT-3.5 given how much cheaper the API pricing is.

So what’s changed? Well, I’m purely speculating here to be very clear, but if GPT-4 is truly a 1 trillion parameter model, then it’s possible that maybe the larger model can just “remember” facts more accurately. Maybe GPT-4 isn’t actually a trillion parameter model and instead is a similar size but the training parameters have resulted in a better ability to produce “correct” results.

Honestly, I have no idea what’s happening, but it’s really interesting to see that this model, which is practically just a more sophisticated autocomplete, can actually “remember” such a wide variety of content. Remember, this isn’t a search engine that’s trying to find the answer. This is a model that’s generating words a few characters at a time that happens to generate tokens that turn out to look like very practical and accurate human reasoning.

We’ll all just have to stay tuned to see what else comes out of this.

Just like last time, I paid for these API requests out of pocket. I sent tens of thousands of requests to the GPT-3.5, GPT-3.5-Turbo and GPT-4 models, this experiment cost me about $759 for all of those requests. I’d be forever grateful if you’d consider buying me a coffee.

So let’s start at the top of the request path. If you go to https://mentallyanimated.com/, which is my website and is hosted on my playground EKS cluster, you go through a bunch of layers. I have a CloudFront distribution which has an AWS NLB as its origin, which points to traefik as my ingress controller, which then routes to an nginx container that has the static assets of the website. At each step in that request path, there’s an opportunity to lose the calling client’s IP address, which is useful for things like rate limiting or allow listing access. Even though the requests start with CloudFront, let’s start with the NLB.

AWS Network Load Balancer

I like using a Network Load Balancer in my stack. It’s highly performant, and a specifically, I like that it doesn’t have any smarts. I can centralize any and all additional middleware features in my traefik configuration, or swap traefik out for any ingress controller and keep any advanced features like rate limits, or path rewriting within the cluster. One catch with it is that it’s layer 4 load balancer. In other words, it doesn’t really understand any concept of things like HTTP headers. It’s just a packet forwarder. So how do we maintain the relevant HTTP headers such that the client IP gets preserved?

As it turns out, NLB’s actually support a configuration setting named Client IP Preservation, which sounds great, right? When creating my kind: Service type: LoadBalancer for traefik, all I’m supposed to need to do is add the right annotation to control the NLB target group attributes.

service.beta.kubernetes.io/aws-load-balancer-target-group-attributes: preserve_client_ip.enabled=true

This alone actually doesn’t work. But why not? It’s because of the way that Kubernetes networking itself works. I’m not going to try to explain it all, but I will link you to this blog post which I read to understand the problem.

To make the preserve_client_ip to work at the NLB level, you need to set the externalTrafficPolicy of your kind: Service to Local, instead of the default Cluster. There’s just one problem with this. If you set that policy to Local, traefik will only ever try to route to pods on the same node as your traffic. If the targeted application pod is scheduled on a different node, traefik will just timeout trying to forward a request. That’s no good. One thing you could do with this is set an affinity rule so that your applications always get scheduled wherever there is an existing traefik pod. Similarly, if you made traefik a kind: DaemonSet and your application also a kind: DaemonSet, then you know that there will always be pods next to each other. These solutions are hacks and there must be something better!

If you look at the documentation for enabling client IP preservation, there’s a fair number of considerations that you have to account for also. One of which seems particularly gnarly.

• When client IP preservation is enabled, you might encounter TCP/IP connection limitations related to observed socket reuse on the targets. These connection limitations can occur when a client, or a NAT device in front of the client, uses the same source IP address and source port when connecting to multiple load balancer nodes simultaneously. If the load balancer routes these connections to the same target, the connections appear to the target as if they come from the same source socket, which results in connection errors. If this happens, the clients can retry (if the connection fails) or reconnect (if the connection is interrupted). You can reduce this type of connection error by increasing the number of source ephemeral ports or by increasing the number of targets for the load balancer. You can prevent this type of connection error, by disabling client IP preservation or disabling cross-zone load balancing.

traefik

Traefik is a very full featured ingress controller. It supports a lot of different kinds of workloads. You can even start using it for HTTP/3, it supports gRPC, etc. One of the things that I learned about along this journey was about something called the proxy protocol. Simply put, the proxy protocol gives the NLB a way to encode things like HTTP header information so that it can be kept and interpreted downstream. Perfect. AWS NLBs also support the proxy protocol, which is still just a simple annotation to add to our kind: Service for traefik.

service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: "*"

And then in my terraform helm_release for traefik, all I had to add to enable proxy protocol functionality was set the following template values.

  set {
    name  = "ports.websecure.proxyProtocol.trustedIPs[0]"
    value = "10.9.0.0/24"
  }

  set {
    name  = "ports.websecure.proxyProtocol.trustedIPs[1]"
    value = "10.9.1.0/24"
  }

These are the private IP ranges for the subnet that my NLB can use, which means we just have to tell traefik that if a request comes from this IP range and is trying to use the proxy protocol, to go ahead and trust those requests.

And that’s literally it. Setting the annotation and setting the trusted IPs made it so that my applications could see the real client IP address. Unfortunately, there’s one more layer to this story.

CloudFront

At the top is CloudFront, which made me think that I was very lucky because only recently did they start supporting forwarding the client IP address. Otherwise, they actually set the X-Forwarded-For header to the IP address of the edge location that the client has connected with.

Previously, IP address and client connection port information were available only in CloudFront access logs, making it harder to resolve issues or perform real-time decision-making based on these data.

There’s a problem with this though. The industry standard is to only check with X-Forwarded-For header when dealing with proxy IP addresses. CloudFront’s solution here is to actually set an entirely new header. You can see these extra headers by using the Managed-AllViewerAndCloudFrontHeaders-2022-06 request policy. They put the client’s actual IP address and the requested port under CloudFront-Viewer-Address. This is both a different header AND a different format since it includes the port information. Traefik and other proxies don’t know what to do with that. You might think to yourself, but Aaron, what if we just modify the headers? I had the same idea and I thought I was being super clever. Separate from a CloudFront Lambda@Edge, I learned that they have even faster and lighter weight CloudFront Functions which run a custom JavaScript runtime, but it’s perfect for modifying headers.

resource "aws_cloudfront_function" "fix_x_forwarded_for" {
  name    = "fix-x-forwarded-for"
  runtime = "cloudfront-js-1.0"
  comment = "Sets the X-Forwarded-For header to the client's IP address."
  code    = <<EOF
function handler(event) {
    var request = event.request;
    var clientIP = event.viewer.ip;

    request.headers['x-forwarded-for'] = {value: clientIP};

    return request;
}
EOF
}

I tried overriding the header before it was sent off to the NLB. Unfortunately, but actually probably fortunately, CloudFront will not forward the request at all when this header is tampered. So that means that I could set yet another header with the correct client IP value, but traefik itself doesn’t let you rewrite headers natively either. I’d have to write a custom plugin or use one of the existing plugins. Maybe Lambda@Edge will also work? That's for another time to try.

Conclusion

So what does this mean? It means that I can’t actually get the client IP address for requests that go through CloudFront. So that’s kind of another L for me this week.

Either way, thanks for reading and see ya’ll in the next one.

That being said, my experiments here didn’t actually pan out very well, but I still wanted to share what I learned. Not every newsletter post has to be a success, right? Especially not when I try to remind myself that this newsletter is more like a public journal.

So what’s the plan? How do I make sure that I scale from 0 more quickly?

Step 1: Make sure that there’s some compute already available

Most of my applications are only configured to use 100 millicores of CPU. That means that we’re really only using 1/10th of a single CPU. So keeping a little headroom around just helps guarantee that the existing pod allocations are leaving enough room on a single node to quickly burst up with some CPU. I use the kubernetes/descheduler to help consolidate and bin pack pods, and since I try to keep nodes as close to 90% utilization at all times, making sure that we don’t overly scale in is big for minimizing scale from 0 time. If I didn’t have any room at all, then the new pod would have to wait for a new node which would take a few minutes.

So what do I do? I schedule some headroom. The concept of headroom is extremely easy. Create a new PriorityClass in your Kubernetes cluster, with a low value of priority, and then schedule an empty pod with the amount of compute that you want to be reserved. If someone makes a request to my scale from 0 service, that pod gets immediately scheduled and the headroom pod with it’s lower priority will get rescheduled at the cluster’s earliest convenience. It’s basically just preallocating the CPU and doing it in a way that makes sure it’s being accounted for in any bin packing logic. Here’s the exact headroom configuration that I use:

apiVersion: scheduling.k8s.io/v1
description: Priority class used by headroom.
globalDefault: false
kind: PriorityClass
metadata:
  name: headroom
value: -1
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: headroom
spec:
  replicas: 1
  selector:
    matchLabels:
      run: headroom
  template:
    metadata:
      labels:
        run: headroom
    spec:
      containers:
      - image: k8s.gcr.io/pause
        name: reserve-resources
        resources:
          requests:
            cpu: 1
            memory: 2Gi
      priorityClassName: headroom

This is extremely simple. The used pause container is an application that does nothing and just doesn’t exit. Defining a priority class is extremely straight forward also.

Step 2: Pre-pull your images

One of the steps for scheduling the pod from 0 is the need to pull the new image to the node that’s going to run it. Depending on your image size, this can take a while. So one thing that I tried experimenting with was pulling the images BEFORE needing to actually schedule any pods, and then making sure that my imagePullPolicy on the containers was set to IfNotPresent. So I did some internet searching to find ways to pull images. I had the idea of maybe just running a docker in docker daemonset that would pull images. I’m pretty sure that this would work but it felt like a maintenance burden that I just didn’t want to deal with. Eventually, I stumbled upon a Reddit post that talked about OpenKruise and its ImagePullJob functionality.

You might be wondering: “But Aaron, didn’t you want LESS maintenance burden?”. I did, but seeing something shiny and new made up for that maintenance burden. OpenKruise has a bunch of other functionality for what feels like native ways to extend and improve a Kubernetes cluster, but I was installing it purely to test out the ImagePullJob.

There was a catch with getting this setup though. There’s no ability for OpenKruise to pull images using the AWS STS token that’s provided by the EKS control plane. This was pretty annoying because it meant that I needed to manually create a docker login token to be used, and I needed to find a way to keep it updated. That’s when I came across this post about running a CronJob to update and save the token. I yoinked the entire implementation and set it up to work in my cluster, along with installing OpenKruise. The entire configuration ended up being about 160 lines in terraform which sounds like a lot but it’s really quite straight forward:

So we create a ServiceAccount that can write secrets to the EKS control plane, and then once every 10 hours, we fetch a new secret from ECR and then store it. That token will be available for OpenKruise to use, so then we install OpenKruise itself.

OpenKruise will run partially as a DaemonSet which is what I was already thinking of doing.

Now, in order to use the ImagePullJob, I simply added the following manifest to the kustomize manifests for each application:

Every time the image tag changed, which it would for every deployment, a new ImagePullJob would get scheduled and OpenKruise would download the specified image in for each node in the cluster. Now I wouldn’t have to wait for the images to be downloaded on that node whenever a page was requested.

Step 3: Reduce the polling interval for pod readiness

As I outline in the scale to 0 post, I settled on using the keda/http-add-on for scaling from 0. And this step is super boring because all I did for this one was set the DeploymentCachePollIntervalMS to a lower number. By default, it checks to see if the deployment has new pods once every 250 milliseconds. I dropped that to 100 milliseconds so that we’d notice more quickly once there was a pod ready to take traffic.

Wrapping up

I was really excited to get OpenKruise to work and start doing the ImagePullJobs. Unfortunately, or maybe fortunately, my images are all extremely tiny. They’re statically linked Go binaries and even though I include the entirety of each set of static assets for the website, the containers only end up being about 30-40 MB even while uncompressed. These take milliseconds for my EKS nodes to pull from ECR. They’re so fast that even after having the images pre-pulled, the containers basically didn’t come up any faster at all. Maybe in the 10s of milliseconds. The ImagePullJobs are also not retroactive, in the sense that they execute one time, at time of image tag update, but if new nodes come up, the image isn’t pulled there. Since I run everything on EC2 spot nodes, my nodes churn fairly often, which nullified the whole point of being able to pre-pull.

What really helped with scale from 0 time ended up being just reducing that polling interval. The containers now get scheduled and respond in about 2.5 seconds, so we’re shaving upwards of a full second off from the page load. Does it really make a difference? No, probably not, but it was fun to try regardless. I think if I had larger images like a Python or Node application, that the ImagePullJob would still be pretty worth it. Alas, that’s not the situation I’m in.

See you for the next one. Thanks for reading

Quick preamble. There’s several levels of experiments here. Maybe most notably, the main contents of this post were actually recorded with my Google Pixel 7 Pro recording app, and then I asked ChatGPT to help correct and fix any punctuation and grammatical errors. I thought it would be interesting to see how well ChatGPT would do with the flaky speech-to-text, and I took my dog on a walk at the same time. I only did some very light manual editing on the content to make up for any random things I said while on the walk!

It's Sunday, February 12th and I'm taking Chopper for a walk. Today, I was hoping to talk about my next newsletter post, as there's a lot to discuss. Recently, I came across a post in the Lenny newsletter about training a chatbot to answer questions based on their content. I found it interesting and wondered how easy it would be to do the same. The Lenny newsletter chatbot used a library called GPT Index. Initially, I looked into fine-tuning my own GPT-3 model, but I didn't have enough content to train it. While some articles suggested that only a couple of dozen examples were enough, I didn't want to go that route. So, I thought of making my chatbot using GPT Index, but I needed content to feed into the model. I thought of writing about myself and my work history.

The first major idea that I thought of was to have a chatbot that was intended for the kinds of conversations that you have with a technical recruiter. With all of the layoffs that have been happening recently, it made me wonder if people would be interested in some tool that would help them skip that first step and try to save everyone some time.

Questions like 'Are you authorized to work in the United States?' could be answered by the chatbot. While this might seem like a frequently asked questions page, most of my projects are experimental and don't necessarily solve a problem, but I don't feel bad about it. So, I started writing a bunch of text about myself, my life, my history, and my work history.

Then, I downloaded the GPT index and ran almost the exact code that was used in the Lenny newsletter.

from gpt_index import (GPTSimpleVectorIndex, LLMPredictor, PromptHelper,
                       SimpleDirectoryReader)
from langchain import OpenAI


def construct_index(directory_path):
    max_input_size = 4096
    num_outputs = 1024
    max_chunk_overlap = 60
    chunk_size_limit = 800

    # define LLM
    llm_predictor = LLMPredictor(
        llm=OpenAI(
            temperature=0.2, model_name="text-davinci-003", max_tokens=num_outputs
        )
    )
    prompt_helper = PromptHelper(
        max_input_size,
        num_outputs,
        max_chunk_overlap,
        chunk_size_limit=chunk_size_limit,
    )

    documents = SimpleDirectoryReader(directory_path).load_data()

    index = GPTSimpleVectorIndex(
        documents,
        llm_predictor=llm_predictor,
        prompt_helper=prompt_helper,
    )

    index.save_to_disk("index.json")


if __name__ == "__main__":
    construct_index("data")

It worked well without any additional information or configuration. That was really interesting, and I decided to understand more about what GPT index does.

GPT index is actually more like a search engine than anything else. It provides several ways to easily source data from your file system or Google Docs, for example, and then convert and run them through the GPT language model embeddings. So what are these embeddings? Simply put, they are numerical representations of words, and since they are numerical, we can perform mathematical operations on them. For every document you have, GPT index will call the embeddings at a point, pass in that document, and then store that in a location of your choosing.

I learned that GPT Index can store the vectors that represent your documents in various databases like Pinecone or Weaviate. But since I did not have a lot of documents, doing everything in memory is completely reasonable. Once we have all of these embeddings then I started to dig deeper.

The vector models use a similarity metric to search for the most similar content from your list of documents. GPT Index has several prompts available that can be used to combine or rewrite the results found from the similarity search. For example:

DEFAULT_REFINE_PROMPT_TMPL = (
    "The original question is as follows: {query_str}\n"
    "We have provided an existing answer: {existing_answer}\n"
    "We have the opportunity to refine the existing answer"
    "(only if needed) with some more context below.\n"
    "------------\n"
    "{context_msg}\n"
    "------------\n"
    "Given the new context, refine the original answer to better "
    "answer the question. "
    "If the context isn't useful, return the original answer."
)

The similarity search is why specialized databases are used, as they are very good parallelizing the computations needed for doing large scale comparisons. Every document in your entire library needs to be compared with the prompts or questions that were passed to you.

Now, you have a working chatbot that can answer questions. The next thing to do is to expose it with a web app so that I could share the link with people.

This brings us to experiment part two. For this post, we used a library called htmx to build the interactivity of the web page. Normally, I build websites in React and TypeScript, but a co-worker recently told me about htmx and I was curious. And since all I needed was a text box and a button, it seemed like a good opportunity to try htmx. Before we get too far into it, I have to say I think I am actually a fan. So what is htmx? htmx is a way to add interactivity to your website where you don't have to write any JavaScript to do the various functionalities of a modern web app. You can add a set of defined attributes to your HTML elements.

You can do things like send HTTP requests and your back end needs to respond with HTML fragments. I have a lot to learn about using htmx, but I was pretty easily able to write my search/chatbot interface to send the question to my back end. In as simple as a few HTML elements, I was ready to send a request to my backend.

<form hx-post="/query" hx-target="#results">
  <label>
    <span>Ask me your own question</span>
    <textarea"></textarea>
    <button type="submit" hx-indicator="#indicator">
      Search
    </button>
  </label>
</form>

<div id="results"></div>

Now, one thing to get into is how did I deploy all of this. I have a set of templates and patterns that I use in a monorepo, which I have laid out in a previous newsletter post. But the whole template pretty much relies on using Go.

However, GPT index is a Python library. One thing that I thought of doing was to take the implementation of the GPT index, or at least the bare minimums for what I was using, and rewrite that in Go. Which, maybe I will do one day.

But for now, at the time of writing, I just wrapped the single library call that I used in the GPT index into a FastAPI application and put all of that into a Docker container. Then I could go back to using a Go-based backend to serve my HTML file. And I would send requests from the Go application to the Python application.

from fastapi import FastAPI
from gpt_index import GPTSimpleVectorIndex
from pydantic import BaseModel

app = FastAPI()
index = GPTSimpleVectorIndex.load_from_disk("index.json")


@app.get("/")
async def root(query: str = ""):
    response = index.query(query)
    if response.response is None:
        return {"error": "no response"}
    return {"response": response.response.strip()}


@app.get("/healthz")
async def healthz():
    return "ok"

Eventually, I got that all wired up and deployed onto my Kubernetes cluster that I use for hosting all of my applications. You can visit <I’ve killed the deployment!> and ask me questions, or rather, ask a question about me.

At this point, I have a functioning MVP where users can ask questions and get answers. The next thing I decided to focus on was experimenting with ways to format my data and documents to best work with the way that GPT index fetches the answers.

Currently, I am using the SimpleDirectoryReader, where each file in a folder corresponds to a single document. The default for similarity is to take the contents of the most similar document in its entirety and rewrite that for the response. For simple questions, that means breaking up my documents into several files that are three to five sentences long. But there is a trade-off because the more documents I have, the more similarity comparisons need to be made to find the most similar document.

On top of that, when GPT index is synthesizing its response, sometimes having the additional context that surrounds the specific sentence or paragraph that answers the question can be useful in crafting a more thorough answer. So, I have to take that into consideration.

Then there is the top N for similarity, aka, how many documents do I take out of the index for crafting the response.

There is another trade-off to consider when splitting information across multiple documents for GPT index. It's important to take into account the combined index and context, which may not be possible unless you increase the number of documents to retrieve. Sharding the data for context like work history can also be a challenge, especially when answering questions about experience with specific technologies. For instance, if a recruiter asks about projects done with Python and the default similarity top N is set to one, the bot would only retrieve a project description from a single job, even though Python was used in multiple projects across multiple jobs.

Now, I need to invert the way that I split my data across documents to get a thorough response. Another possible idea is to duplicate the information into multiple documents and hope that the similarity search picks the document where I've repeated information. For example, I can have documents for each job and also have documents that are specific to describing experience with a single technology, such as a text file with descriptions of projects I've done in Python. I don’t think I found the exact right balance in my time frame, but I think there’s a lot of potential.

I'm still very curious about what kinds of results I could get if I fine-tuned the general GPT-3 model. One benefit of this search style approach is that it minimizes the chance of hallucinations in the model, aka answering questions with information that is not true or is not in the index. This chatbot only surfaces preexisting information. So I shouldn't be too worried about it making ridiculous responses.

I shared this URL with a couple of friends of mine, and it's no surprise every single one of them, including my wife, came up with questions to ask it that I did not prewrite. That isn't all that unexpected. But it was still a fun project nonetheless.

What do you think, dear reader? Do you think personalized FAQ style chatbots are worth it? If you’re searching for a job, do you think you would ever send a recruiter a message linking to your own resume chatbot? And if you are a recruiter, how do you feel about being given a link like this to ask all your questions too? I would love to know what you think.

Anyways…

I’ve come to learn that most of my ideas are really bad. In my mind, I keep hoping that some project of mine is going to go kind of viral and like, maybe I become famous for it or something. In reality, I know that banking on any one particular site to do well isn’t great. On top of that, my projects are always pretty small. An old co-worker of mine said that I have “one weekend syndrome”. I’m only interested in a project for about a weekend before I move on, which is… it’s pretty true. It’s hard to build an actual product that people would pay for in a single weekend.

Here’s the other catch, I also think of these projects like going out on a fishing boat. I’ve gone fishing twice in my life. One of those times was with my father-in-law in Mexico where we went out on a boat that had like 12 fishing rods sticking out of it, even though there was only 4 humans on the boat. How does that relate to my side projects? I know most of my ideas are bad, so instead of worrying about how bad they are, I ignore it completely and think that if I put out as many as I can, then maybe one of them will actually stick. Angry Birds was the 51st game made by Rovio. That’s a lot of games. And now they have a whole ass movie and stuff. BUUUUUUUT, I don’t have any funding. I don’t have angel investors or VCs funding anything. So I need to pay for everything and that can get pretty expensive.

A lot of hosting websites will give you a free tier of some kind and that’s great, but the free tier tends to only be useful for one project worth of compute unless you do something that’s a little cheeky.

Heroku’s basic tier costs about $5 a month. Fly.io gives you about $6 a month worth of compute on the free tier. The smallest DigitalOcean VM is still going to run you $4 a month. Vercel is free but only for personal use and not commercial use. AWS has a pretty cool sounding free tier. For the first 12 months you get things like 750 hours a month worth of compute for free if you run the smallest possible instances. But it’s pretty well understood these days that the AWS free tier is delightfully deceptive. And you only get one of those EC2 instances for free. If you run two for the month, you’re going to pay $7.60 a month for that second t3.micro. If you’re comfortable with AWS, you can run it as a spot instance and that’s still $2.27 a month.

If I had a bunch of side projects, this all starts to add up really quickly. But let’s be real. My AWS bill is something like $210 a month. $75 a month for my EKS control plane. About $30 a month for 3 t3.medium EC2 instances. About $55 a month for my Aurora Postgres cluster, and about $40 a month on CloudWatch logs. The rest is in small storage costs like ECR or S3. Yes, yes. I know about things like serverless Aurora. I know that I could reduce how much logging I’m doing.

So scaling to 0 isn’t exactly going to save me that much money. That’s not the point though, I like messing with the infra that I have and hey, it’s my experiments. I get to do what I want. I consider it an investment in myself and my growth because I have an amazing time just messing around with all of the things in my cluster. I like the Kubernetes ecosystem. Tons of software is available these days as a docker container and sometimes even a helm chart, which means it takes literally minutes for me to stand up a new project to play with in some cases. Fortunately, that amount of money is probably going to stay pretty flat for as long as I keep making projects. So if I come up with like 40 ideas, I might actually break even. Scaling to 0 helps with that so we’re going to talk about it.

Yeah, good point. So when I’m talking about scaling to 0, I’m talking about making sure that if a project or website that I’m running isn’t actually being visited by anyone, that the code isn’t actually running anywhere at all. Scaling on demand, serverless, functions as a service. These are some of the other names that I’ve heard used for this. It’s a really interesting concept. But SOMETHING has to run, right? Yep, so what am I running? How am I accepting requests to my EKS cluster and how is it knowing to schedule pods?

OpenFaaS

OpenFaaS is a functions as a service framework that you can deploy to a Kubernetes cluster. Also maybe worth trying one day but it basically reinvents how you do things like deployments. You use their CLI for building and deploying functions and I wanted to keep things more Kubernetes native, which actually leads me to…

KNative

KNative is probably the default that most people end up coming across. I haven’t actually tried it but thought I should mention it. I’ve purposefully decided that I wanted to try to avoid KNative entirely if I could, because in order to use it, you have to install a separate networking layer. That’s a little more complexity than I care to actually try right now, but maybe one day.

Sablier

Sablier is actually the first project that I tried. And conceptually, I actually REALLY like how it works. When I set it up, I configured it as a plugin for Traefik which is my ingress controller of choice these days. I got it set up in about 25 minutes and started to play with it. The way it works is that you set it up as Traefik middleware, which means when Traefik receives a request, before it makes it to your service, it goes through sablier first. The way sablier itself works is that when the middleware is triggered, it sends a blocking request to the sablier service that you install into your cluster. When the middleware sends a request to sablier, sablier checks to see if it needs to change the number of replicas available.

Sablier has a few different modes. It has a “dynamic” mode which will immediately serve a temporary web page that you can customize like so:

The temporary page is setup to auto-refresh itself until the actual application is up and running. I really like this feature. It’s great for any website that has static content. The other mode is called “blocking” and unfortunately, this is where I ran into a big problem. Blocking mode waits before forwarding the request. The first request for a “blocking” mode request will actually fail, which I’ve reported but I can’t seem to find a good fix for. There are some other interesting oddities about sablier. Namely, you configure it by having a specially formatted name. Like… there’s already other configuration in this project, why am I configuring things like the Kubernetes namespace of the scale target by joining a string with underscores?

keda-http-addon

The project I ended up sticking with is the keda-http-addon. It’s also an experimental project but I’ve been wanting to play with its core KEDA component for a while. KEDA itself is a set of abstractions for managing event based autoscaling and it’s a lot more flexible than the build in horizontal pod autoscalers. You can use things like SQS queue attributes as your scale metric, or use database queries to trigger your scale out events. The HTTP addon works as a proxy that queues up HTTP requests. So you configure your ingress controller to send all requests to the “interceptor” and the interceptor holds the requests while your applications scale out. This has a different problem from sablier though which I still need to figure out. The first request to the HTTP addon takes like 25 seconds to respond, even though the application has been running after 3-4 seconds.

BUT, the request doesn’t fail completely which is a big deal. For applications like CatStories.ai, it receives webhooks from Twilio when you send text messages. Having the first request fail like with sablier’s “blocking” mode is not going to be great for that use case.

Closing

Another totally random thing to consider. EC2 instances have a max number of IP addresses that it can have. That’s actually why I don’t use smaller EC2 instances for my cluster. So if I have too many applications running, even if they don’t use much actual compute, I may be forced to spin up another EC2 instances due to IP address exhaustion. There are ways around this too by using separate CIDRs and what not but I don’t have those configured… yet.

That’s all I’ve got for this week. Maybe a less exciting experiment than some of the other things I’ve written about but that’s what I played with. Now some of my really low traffic projects like https://terragen.sh and https://catstories.ai might take 25 seconds to load but the sites are up when people want to check them out but they’re not eating up resources when no one is checking them out. And I can keep the links alive on my various internet bios and stuff.

About 6 years ago, I had the idea to see how well you could answer trivia questions with AI. I used word2vec and picked answers based on which multiple choice answer had the highest cosine similarity to the average of the vector of the sentence. Simply put, this… didn’t work at all. The percentage of questions that this method got correct was not notably better than selecting answers at random.

When the word2vec paper first came out, it was revolutionary that we could do analogies for the semantic meanings of words. “King - Man + Woman = Queen”. I was floored. The original paper came out in 2013, and here we are in 2023. A whopping 10 years later and countless generations of advancement in machine learning, comprehension, understanding, etc. In June of 2017, we were blessed with the “Attention is all you need” paper which introduced the transformer architecture that was the step function that has brought us this revolution in natural language processing. So, what’s different now?

In late 2022, OpenAI released ChatGPT. Which, at the heart of it all, is a brilliant UX that someone came up with for interacting with GPT-3.5. The original GPT-3 was made available in the summer of 2020 in private beta, and while there’s a playground UI, usage of GPT-3 was still fairly intimidating for the every day person. The playground featured all the knobs and parameters for tuning GPT-3’s response and that turned off many people from using it. ChatGPT on the other hand was… a chat bot. It looked no different than talking to someone in Slack or Microsoft Teams. Anyone could sign up, there was no wait list, and no scary knobs. ChatGPT gained 1 million users in only 5 days. For the rest of this article, I will refer to the model as ChatGPT, even though that’s not perfectly accurate since ChatGPT is a fine-tuned version of GPT-3.5.

So, how smart is it? 6 years later from my original experiment, I wanted to find out what the capabilities are of some of the most impressive machine learning we’ve seen to date. We’ve seen the creative works of ChatGPT. I even made a fun website for my wife and I to use: https://catstories.ai. ChatGPT is amazing at creative tasks, but for tasks that involve more knowledge, the opinions are significantly more split. Some people are convinced that ChatGPT, with it’s intuitive interface, is going to replace traditional search engine’s like Google. Microsoft is integrating ChatGPT directly into its search engine, along with other products.

ChatGPT is taught everything it knows by reading vast amounts of the internet, and it uses that information when it generates text. For the non-technical, you can think of it like your phone’s autocomplete, but way smarter. If you’re clever about how you prompt ChatGPT, you can get it to do revolutionary things and test its knowledge. For example, ChatGPT was able to pass the Bar exam. You can have it write marketing copy, or write emails for you. But if we’re going to use it as a source of truth to replace things like search engines, how does it do with general knowledge?

How did I test it?

The uberspot/OpenTriviaQA dataset is a creative commons data set of multiple choice questions. The questions are split into 22 categories such as: animals, entertainment, history, movies, and sports. Most of the questions have 4 possible multiple choice answers, but some of the questions are true or false questions.

Questions come in the following format:

#Q What is the name of Rocky Balboas restaurant?
^ Adrians
A Rockys
B Mickeys
C Adrians
D Apollos

First, you have the question itself. Then you have the correct answer. Lastly, you have your options for answer, one on each line.

My script would download each of the categories and ask each and every question to ChatGPT. For the above question, I would format my prompt to look like so:

I am a highly intelligent multiple choice trivia bot. You are given a multiple choice question. You must choose the correct answer from one of answers. Only include the answer and nothing else.
Question:
What is the name of Rocky Balboas restaurant?

Possible answers:
Rockys
Mickeys
Adrians
Apollos

Your answer:

I would then send this to ChatGPT and compare its answer with the correct answer.

Soooooo, how’d it do?

Here’s the raw data, sorted from least to most correct:

Category        Correct Total   Percentage
brain-teasers	103	207	0.497585
video-games	310	599	0.517529
television	2911	5230	0.556597
entertainment	163	280	0.582143
animals		815	1366	0.596632
celebrities	1909	3196	0.597309
sports		1728	2840	0.608451
movies		2647	4314	0.613584
for-kids	485	759	0.638999
music		3746	5579	0.671447
literature	888	1288	0.689441
hobbies		867	1242	0.698068
general		2306	3290	0.700912
newest		2117	3016	0.701923
people		1974	2743	0.71965
technology	1820	2486	0.7321
world		3571	4875	0.732513
religion-faith	469	638	0.73511
history		1228	1645	0.746505
rated		1640	2185	0.750572
humanities	831	1097	0.75752
geography	652	842	0.774347
overall 	33180   49717   0.667377

It’s really interesting to see how these categories breakdown. Notably, ChatGPT did not do well with the brain-teasers category. Let’s look at a few examples of these questions.

#Q The name of what is coded in the following: 86,smopuim?
A it does not mean anything
B the name of a software product
C a NASA aircraft
D a secret CIA project

This one took me a while to figure out but ChatGPT got it correct. The answer is B. Those characters say windows,98 upside down. When I was talking to a friend of mine over at Fearless AWS, he suggested that I try to get ChatGPT to explain itself. So I did that.

I am a highly intelligent multiple choice trivia bot. You are given a multiple choice question. You must choose the correct answer from one of answers. Only include the answer on the first line. On the next line, explain your answer.
Question:
The name of what is coded in the following: 86,smopuim?

Possible answers:
it does not mean anything
the name of a software product
a NASA aircraft
a secret CIA project

Your answer:
the name of a software product
The numbers 86 and smopuim are likely a code for a software product, as they are not a recognizable acronym or phrase.

At first, I was impressed that ChatGPT was able to figure this out but it actually seems like ChatGPT just guessed.

Let’s look at an example where ChatGPT was incorrect.

I am a highly intelligent multiple choice trivia bot. You are given a multiple choice question. You must choose the correct answer from one of answers. Only include the answer on the first line. On the next line, explain your answer.
Question:
What number multiplied by 10 equals the square of the same number, times 5?

Possible answers:
4
5
2
10

Your answer:
5
Explanation: 5 multiplied by 10 is equal to 50, which is the square of 5, times 5.

Here’s a great demonstration of how ChatGPT can be pretty dumb. It got arithmetic completely wrong. Because it just generates text, that doesn’t mean that it has correctly encoded arithmetic. The correct answer here is 2. 2 squared is 4, times 5 is 20. 2 times 10 is also 20.

Let’s look at another example where ChatGPT was correct according to the data set, but the explanation doesn’t quite add up:

I am a highly intelligent multiple choice trivia bot. You are given a multiple choice question. You must choose the correct answer from one of answers. Only include the answer on the first line. On the next line, explain your answer.
Question:
A man is carrying three croquet balls across a bridge. The bridge has a sign that says bridge can only hold 200 pounds. The man weighs 195 pounds and the croquet balls weigh 2 pounds each. The man ends up crossing the bridge in one trip and no one else helps him. This is possible.


Possible answers:
False
True

Your answer:
True
The man is carrying three croquet balls that weigh 2 pounds each, for a total of 6 pounds. This means that the man and the croquet balls combined weigh 201 pounds, which is less than the maximum weight of 200 pounds that the bridge can hold. Therefore, the man is able to cross the bridge in one trip.

So, this is a really interesting one. The data set doesn’t have an explanation for any of these brain teasers but it lists the correct answer as “true”. If you do some googling, you’ll find that the answer of the riddle is supposed to be “true” because the man can juggle the balls. Ignoring the actual physics of such a situation, the ChatGPT answer, once again, gets the arithmetic completely incorrect in saying that 201 pounds is less than the maximum weight of 200 pounds.

Let’s look at one final example that addresses something in the NLP field called the Winograd schema. Which, to over simplify, is basically being able to understand how pronouns resolve to ambiguous actors in the same sentence.

I am a highly intelligent multiple choice trivia bot. You are given a multiple choice question. You must choose the correct answer from one of answers. Only include the answer on the first line. On the next line, explain your answer.
Question:
If the mother of a boy spanks him and then his father comes and does the same, who hurts the most?

Possible answers:
The boys brother
The mother
The boy
The father

Your answer:
The boy

Explanation: The boy is the one being spanked, so he would experience the most pain.

In this answer, ChatGPT explains that the boy is the one being spanked and so even though there might be some ambiguity between the usages of “him” and “his” referring to either the boy or the father, ChatGPT understands the context of the boy being spanked.

These brain teasers require logical conclusions that it seems like ChatGPT doesn’t really reason through. It makes sense that this is the category that ChatGPT has failed most on, but there are also some impressive answers. That being said, even asking ChatGPT to explain itself is a generated response. There’s nothing to say that this is legitimately how ChatGPT is thinking about the problems.

Let’s look at another category, literature, which presumably ChatGPT should be good at because there should be plenty of text and references to learn about the literature.These brain teasers require logical conclusions that it seems like ChatGPT doesn’t really reason through. It makes sense that this is the category that ChatGPT has failed most on.

I am a highly intelligent multiple choice trivia bot. You are given a multiple choice question. You must choose the correct answer from one of answers. Only include the answer on the first line. On the next line, explain your answer.
Question:
In what book of the Harry Potter series do Harry and Ginny kiss for the first time?

Possible answers:
Harry Potter and the Order of the Phoenix
Harry Potter and the Prisoner of Azkaban
Harry Potter and the Deathly Hallows
Harry Potter and the Half Blood Prince

Your answer:
Harry Potter and the Half Blood Prince

Harry and Ginny kiss for the first time in Harry Potter and the Half Blood Prince, when Harry visits Ginny in the Gryffindor common room.

In this case, ChatGPT is correct. Harry Potter is popular enough with a large fandom that it makes sense that this is probably talked about alot.

I am a highly intelligent multiple choice trivia bot. You are given a multiple choice question. You must choose the correct answer from one of answers. Only include the answer on the first line. On the next line, explain your answer.
Question:
Harry Potter marries Hermione in the last book of the series.

Possible answers:
True
False

Your answer:
True

Explanation: In the epilogue of the final book, Harry and Hermione are married and have two children.

Another really interesting case here. Harry marries Ginny and has two kids, not Hermione. Trying to think through how this might have happened, I wonder if internet comments and maybe fan fiction that was part of the training set had enough content that preferred Harry and Hermione to be together. The implication there for the greater knowledge base of ChatGPT is that you could influence what it deems to be true if there’s enough of an *opinion* in its training data.

Wrapping up

Like I said, there are almost 50,000 questions. Digging into each and every single one of them is not practical for a single post, but it is really interesting to try to intuit about why ChatGPT is better or worse at various categories. In categories like brain-teasers, there’s much more logic than needs to be applied and combined to answer some of the questions. In other low scoring categories like television and video games, my guess is that because the content itself is not in an indexable / consumable format, ChatGPT doesn’t have as much information about it. ChatGPT hasn’t played the games themselves or watched the television shows. There might be 3rd party content ABOUT what happened in the shows but the contents of the show would be mentioned in passing instead.

In high scoring categories, the categories all tend to be for contents that have been documented in text long before we had audio, video, etc. The geography category and the history category have content that has existed long before we had the technology and storage to regularly keep things like television content.

All that being said, a 66% correctness is still pretty impressive given such a vast selection of topics. But as we learned, sometimes the answers are reportedly correct but the explanations are not always correct. I don’t think ChatGPT or any large language model that we have right now is the strong AI or general AI that some articles want you to believe it is. I don’t think any large language models that are generative in nature should replace traditional search engines that surface high quality information. There are search engines like you.com that are exploring the intersection but it’s still early days for this technology!

Between development and doing the final runs, this experiment cost me about $100 to hit the GPT-3 API for all the questions. I’d be forever grateful if you’d consider buying me a coffee. If a company would be willing to sponsor the work, it’d be interesting to try different prompts, randomizing the orders of the answers, try running the tests multiple times, etc. I’d also love to run the same test over the full Jeopardy question set, but given how expensive this test was, running the test over a question set almost 10x as much would be too much for me to do right now.

As always, the code that I wrote for this is all available. Check out this post’s code, including the answers of ChatGPT, by clicking here.

A few months ago, I migrated my newsletter to substack and the first post I published here was about using S3 as your remote layer cache. When that post was published, a kind user on Twitter told me about https://depot.dev. One of the founders of Depot responded to the thread and it got me really curious about trying the platform.

A slice of experiments

How to use S3 for your docker layer cache

As of May 2022, the buildkit backend for building containers has experimentally supported using S3 as a backend cache. For over 2 years, there’s been no other way to store max mode container manifests in AWS, as ECR still doesn’t support cache manifests. On the bright side, in October 2022, we finally started getting some communications from AWS about s…

Read more

4 years ago · 1 like · Aaron Batilo

I got signed up and started building my containers and I was blown away. Since they support GitHub OIDC for authentication and since they’re a drop in replacement for docker build, getting setup genuinely took a few minutes. I had to switch off using my current bake file approach since Depot doesn’t currently support bake (but they’re interested in supporting it!) but even then, running all of my docker builds in parallel via a GitHub Actions matrix, my builds went from about 3.5 minutes down to about 50 seconds in the cold case and about 15 seconds when a given container in my monorepo hadn’t changed.

I was flabbergasted. I became obsessed with understanding how the improvements were so ridiculous. The Depot founders were not shy about sharing some of the secret sauce. I spoke with them directly but they clearly explain the improvements right in their documentation.

Depot is a remote container build service that makes image builds 3-14x faster than building Docker images inside generic CI providers. Docker image builds get sent to a fast builder instance with a persistent cache. The resulting image can then be downloaded locally or pushed to a registry.

The persistent cache! The SSD that they attach to their builder instances holds a cache that makes everything significantly faster. Having the cache right on disk means that you don’t have to spend any time transferring any cache artifacts from a remote location. Depot themselves have a well documented architecture for how they’ve implemented their builders. It’s isolated, secure, and managed. They support a flurry of other features, but I wanted to figure out how to simulate their setup for my own learning and now I’m here to share it with you, and to share what additional features are available when you know that you have a persistent cache.

To be clear, I’m not officially affiliated to Depot in any way whatsoever. This is not a paid post. I’m just genuinely impressed and fascinated by their product.

Who is this for?

This post is written with the assumption that you already have some experience with Kubernetes and with AWS. The concepts and examples should translate fairly directly to other clouds or other managed Kubernetes offerings

Running Docker buildkit in Kubernetes

As it turns out, Docker buildkit already supports being able to run in Kubernetes via its “Kubernetes driver”.

⇒  docker buildx create \
  --bootstrap \
  --name=kube \
  --driver=kubernetes \
  --driver-opt=[key=value,...]

This command will quickly use your current Kubernetes context and create a remote buildx agent that you can use to build your containers. This creates all of the relevant Kubernetes resources for you automatically. This method creates a kind: Deployment in your cluster that you can scale out. The —driver-opt that are documented in the link above will let you control things like number of replicas, and CPU/memory allocations.

Of course, if you want even more control, you can refer to buildkit’s list of examples in their GitHub repo. They have examples for authenticating with TLS, examples of using other deployment types in your Kubernetes cluster, etc. For this article, we’re going to break it down step by step and we’re going to rely on the Kubernetes cluster authentication for accessing the remote agents. From inspecting the resource that using the docker command gives us, we can come up with the following minimal Kubernetes deployment.

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: remote-buildkit-agent
  labels:
    app: remote-buildkit-agent
spec:
  replicas: 1
  selector:
    matchLabels:
      app: remote-buildkit-agent
  template:
    metadata:
      labels:
        app: remote-buildkit-agent
    spec:
      containers:
        - name: buildkitd
          image: moby/buildkit:buildx-stable-1
          readinessProbe:
            exec:
              command:
                - "buildctl"
                - "debug"
                - "workers"
          securityContext:
            privileged: true

That’s it. That’s the bare minimum you need to schedule a remote agent. Now we need to figure out how to connect to it so that we can start to send it work to do.

As a quick aside, you’ll get a massive speed up for pushing your docker images, if you run this remote buildkit agent in the same cloud region that you’re storing your images. I run my EKS cluster in us-west-2 and all of my ECR repositories are also configured for us-west-2. If you’d like to read more about this speed up, I talk about it in a previous newsletter post where I got almost a 30% improvement.

A slice of experiments

Pushing big docker images more quickly using docker buildx and your EKS cluster

Building containers is easy when your applications are small. I’m usually doing projects in Golang, but at work we’re a monolithic Django application and after we install all our system dependencies and application dependencies, we end up with a 4.5GB container. It takes a while to build and even longer to push the container. Because of that, I started …

Read more

5 years ago · 1 like · Aaron Batilo

Authenticating with remote buildkit using a kubeconfig

When you run the docker buildx create command, a file gets created on your computer with the buildx authentication configuration. If you use the following command:

⇒  docker buildx create --name remote-buildkit-agent --bootstrap --use --driver kubernetes

Then your configuration will be created like so:

⇒  cat ~/.docker/buildx/instances/remote-buildkit-agent | jq
{
  "Name": "remote-buildkit-agent",
  "Driver": "kubernetes",
  "Nodes": [
    {
      "Name": "remote-buildkit-agent0",
      "Endpoint": "kubernetes:///remote-buildkit-agent?deployment=&kubeconfig=",
      "Platforms": null,
      "Flags": null,
      "DriverOpts": null,
      "Files": null
    }
  ],
  "Dynamic": false
}

This file will be named after the agent that you created. The file you see above is what gets created for you automatically, but we can remove most of the fields and everything will still work. All you actually need is:

⇒  cat remote-buildkit-agent | jq
{
  "Name": "remote-buildkit-agent",
  "Driver": "kubernetes",
  "Nodes": [
    {
      "Name": "remote-buildkit-agent",
      "Endpoint": "kubernetes:///remote-buildkit-agent?deployment=&kubeconfig=",
      "DriverOpts": {
        "namespace": "default"
      }
    }
  ]
}

Do take note that with this configuration, the kubeconfig field is blank which means that the docker client will look for your kubeconfig at the default location. By applying the minimal deployment file and creating this configuration file manually, you’ll be able to start using the remote agent already.

docker buildx use remote-buildkit-agent

Now every time you execute docker buildx build, you’ll execute the build remotely. Your local CPU and memory won’t be used at all. This also means that if you specify a larger CPU or memory allocation in the deployment, that you’ll be able to leverage significantly more powerful build machines. That alone could help speed up your docker builds actually, and that’s even before we get to any of the caching optimizations.

Configuring a more generous garbage collection policy for buildkit

If you build some image a few times with this existing remote agent, you’ll probably notice that a lot of your steps get nicely cached, but then quickly stop being cached when you build another image. This is because buildkit keeps only a limited amount of space for its cache. Fortunately for us though, we can modify that limit and have garbage collection happen less often, which makes the cache much more useful for us.

If you look at the buildkit GitHub repo, we can find some documentation and examples for the default buildkit configuration. This documentation does a great job on its own for all of the other config options, but the sections that we really care about here are the *.*.gcpolicy rules. These sections are tiered and allow for multiple levels of eviction policy. I will make one quick shout out for the max-parallelism option which you might want to leverage when you have a large number of CPUs allocated to your buildkit agent.

Let’s take a look at one of the sections in the documented example:

  [[worker.oci.gcpolicy]]
    keepBytes = 512000000
    keepDuration = 172800
    filters = [ "type==source.local", "type==exec.cachemount", "type==source.git.checkout"]

  [[worker.oci.gcpolicy]]
    all = true
    keepBytes = 1024000000

First we see the sections keepBytes which is the number of bytes you want to allocate to the matching cache. 512000000 bytes, aka 512MB is being allocated for specific types of artifacts. There’s the local cache for source code that’s a part of the docker context, there are cache mounts (which we’ll talk more about in the next session), and artifacts for a remote git checkout (like when you run docker buildx build on a public GitHub URL). At the same time, you can also set a time based eviction. That’s what keepDuration is for. The value 172800 is in seconds which translates to 48 hours. In this case, evictions will happen with whichever case happens first.

The second gcpolicy has the all key which will be the fallback. 1024000000 bytes is 1GB of space for all other caches and cache types. If you have a large number of dependencies in your build, you can easily spend more than 1GB on space.

This is the big win that will drastically speed up our builds. Let’s get into how we integrate that in with our existing Kubernetes deployment. Ultimately, this consists of making two changes. The first being that we need to allocate an EBS volume to store our actual cache, and then we need to update the config that our remote agent uses and mount that to the buildkit container.

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: remote-buildkit-agent
  labels:
    app: remote-buildkit-agent
spec:
  replicas: 1
  selector:
    matchLabels:
      app: remote-buildkit-agent
  template:
    metadata:
      labels:
        app: remote-buildkit-agent
    spec:
      containers:
        - name: buildkitd
          image: moby/buildkit:buildx-stable-1
          volumeMounts:
            - name: config
              mountPath: /etc/buildkit
            - name: var-lib-buildkit
              mountPath: /var/lib/buildkit
          readinessProbe:
            exec:
              command:
                - "buildctl"
                - "debug"
                - "workers"
          securityContext:
            privileged: true
      volumes:
        - name: config
          configMap:
            name: remote-buildkit-agent
        - name: var-lib-buildkit
          persistentVolumeClaim:
            claimName: remote-buildkit-agent
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: remote-buildkit-agent
data:
  buildkitd.toml: |
    root = "/var/lib/buildkit"

    [worker]

    [worker.containerd]
      enabled = false

    [worker.oci]
      enabled = true
      gc = true
      gckeepstorage = 30000000000
      snapshotter = "overlayfs"

      [[worker.oci.gcpolicy]]
        filters = ["type==source.local", "type==exec.cachemount", "type==source.git.checkout"]
        keepBytes = 10240000000
        keepDuration = 604800

      [[worker.oci.gcpolicy]]
        keepBytes = 30000000000

      [[worker.oci.gcpolicy]]
        all = true
        keepBytes = 30000000000
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: remote-buildkit-agent
spec:
  resources:
    requests:
      storage: "50Gi"
  accessModes:
    - "ReadWriteOnce"

Take all the contents of the above yaml and assuming you have a CSI provisioner configured on your cluster, this will allocate a 50Gi persistent volume, and then we attach that to the remote-buildkit-agent container, and we also create a ConfigMap with a minimally configured buildkit file that will increase the garbage collection thresholds. In this example, the buildkit config file is configured for a 30Gb persistent cache. That means we can hold significantly more layers in the local cache and layers that haven’t been touched will still get evicted as we fill up the cache.

Leveraging mount caches with persistent disk buildkit

Newer versions of the buildkit backend support an entirely different kind of cache called the run cache mount. Anywhere that you specify a RUN command in your Dockerfile, you can add a cache mount that lets you specify directories that you’d like to cache as the type==exec.cachemount artifact in your cache. For example:

RUN \
    --mount=type=cache,target=/var/cache/apt \
    apt-get update && apt-get install -y git

This stores the /var/cache/apt directory into a different type of cache that’s explicit, instead of having the generic layer cache store the results of the apt commands.

As opposed to the buildkit inline cache, or the existing remote caches for remote registries, you won’t see as much documentation about this time of cache yet both because it’s a new feature, but also because there isn’t a build provider in the world that can leverage this type of cache except for Depot. This is because providers like GitHub Actions don’t give you dedicated disk space that you can use. GitHub Actions and CircleCI, etc, all let you cache directories on the host within some limit, and there’s an open issue for buildkit to be able to specify the cache directory that gets used. However, at the moment, either you run your own remote agents or you use Depot to get persistent disk. Having a run cache mount is a phenomenal speed increase if your applications can use incremental compilation of some kind.

For example, the Go toolchain introduced a package build cache in 1.10. That means that you can have a Dockerfile line like so:

RUN \
  --mount=type=cache,target=/root/.cache \
  --mount=type=cache,target=/root/go/pkg/mod \
  go test -v ./... && \
  CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go install -ldflags="-w -s" ./cmd/...

If you run your tests at the same time as your build like I outlined in my previous newsletter post, then the built test artifacts and results will also be leveraged with this run type. This makes running your tests and building your actual artifacts as fasts as possible because you’ve minimized re-work.

For even more examples of using a run cache mount, I’d recommend that you check out this article by Vladislav Supalov.

Wrapping it all into a GitHub Action Workflow

Combining some of the steps from earlier in this post, you can authenticate a GitHub Action Workflow just by writing a kubeconfig and the listed docker config. Normally, you would use the docker/setup-buildx-action action with the right driver to get a buildkit agent in your cluster. Unfortunately, if you have already created the agents like you would with the custom config, this action doesn’t let you connect to existing agents. So you’ll need your own version of the same command. With EKS, I leverage the aws eks update-kubeconfig command and then write the json file myself. The next big key here is the docker buildx use command.

      - name: Connect to remote buildkit agent
        env:
          REMOTE_AGENT_NAME: remote-buildkit-agent
        run: |
          aws eks update-kubeconfig --name your-cluster-name
          mkdir -p ~/.docker/buildx/instances/
          cat << EOF > ~/.docker/buildx/instances/"$REMOTE_AGENT_NAME"
          {
            "Name": "$REMOTE_AGENT_NAME",
            "Driver": "kubernetes",
            "Nodes": [
              {
                "Name": "$REMOTE_AGENT_NAME",
                "Endpoint": "kubernetes:///$REMOTE_AGENT_NAME?deployment=&kubeconfig=",
                "DriverOpts": {
                  "namespace": "default"
                }
              }
            ]
          }
          EOF
          docker buildx use "$REMOTE_AGENT_NAME"

If you set the docker buildx context correctly, then if you use the available docker/buildx-push-action, you’ll automatically use your remote buildkit agent with your persistent cache. You could even use docker/bake-action with this setup since it’s native buildkit. This is actually one of the advantages of using the self-hosted option over Depot’s offering.

What’s next?

The EBS persistent volume that we used in this post can only be attached to once pod replica at a time. That means that we can’t scale out our build agents. Even if you increase the number of replicas for the deployment, the PVC will only attach to one of the replicas. One way that you can improve this is by using a Kubernetes StatefulSet. But now you’ll have multiple copies of the caches which are not guaranteed to be the same. The Kubernetes buildkit driver does have a parameter called loadbalance which you can leverage for some amount of predicability but not much. Something that I do want to try experimenting with in the future is using an EFS mount in EKS so that I can attach the same filesystem to multiple replicas.

Another problem with our configuration is that we currently run the buildkit agent as a privileged container. From a security perspective, it’s not ideal that there are additional privileges. There are ways to run the buildkit agent with rootless mode being set to false but that’s for another time.

The last big piece of functionality that is missing here is any configuration for doing cross platform docker builds. For Go applications, cross compilation is very easy, but for some application stacks, it’s significantly easier to build natively on other platforms.

Just for the final piece of thoroughness, many of the problems that this custom, self-hosted configuration has, are solved by the folks doing Depot. Again, I’m not officially affiliated with them at all. Just a genuinely big fan of their product.

That’s all for now, folks. This ended up being quite a long post! As usual, sample files are also available on GitHub. Let me know what you think and I’ll see you next time. Please consider subscribing to get future posts, and consider sharing with your friends!

Method #1: Don’t. Run your tests directly in CI hosts

This might be everyone’s default approach. Organizations that are still new to using containers in general might only be interested in doing so for your final deployment. In this pattern, you do things like run your tests and build your release artifacts in a separate environment, and then copy them directly into your Dockerfile.

Pros:

• Arguably the most straightforward

• Plenty of examples and documentation for how to do this since it’s the default for a lot of people

• Easiest to adopt in transition to Dockerfiles

Cons:

• Not portable. Coupled directly to CI provider

• Need to maintain your CI environment on top of other environments

There is plenty of documentation for how to do this. Installing things like Python in your CI environment is going to be heavily supported and widely adopted. This method is easy to get started with. However, some of the cons are incredibly inconvenient. Relying on your CI vendor means that you’ll probably end up integrating with a lot of their proprietary functionality which means vendor lock in. Solving that problem is what systems like https://dagger.io and https://earthly.dev are trying to do. Additionally, you’ll notice that this workflow requires you to install python in your CI environment. You also have to have python installed in your deployed environments. This means you have a bifurcation in how you manage and administer your CI environment and how you manage and administer your live environments.

In this example, you’re only installing Python, but in real environments with useful applications, you need to install more complicated dependencies. In the Python community, it’s very normal for libraries to rely on or expect operating system level dependencies to be installed. In languages like Go, you can have all dependencies be statically linked, but in Python, if you want to communicate with a Postgres database, almost all of the popular libraries expect you to have an operating system dependency like libpq-dev to be installed. Now you need to install libpq in your CI environment and your live environment. And this problem balloons. If you want to be truly confident, you need to coordinate that when you update your CI environment, that you also update your live environment at the same time. If you don’t, you risk that you’re actually running your tests with different versions of dependencies than your live environment, which means you might get different behaviors in each environment.

Method #2: Run your tests on a pre-built CI image

Many CI services will let you specify the container to actually execute your CI steps in. In GitHub Actions, this is the container key. In my example, I’m just using the official python image, but you could separately build a CI image that has all of your prerequisites.

Pros:

• Decreasingly less coupled to your CI provider, relative to method 1

Cons:

• Requires a separate CI process for the base image that you want to use

In method 1, we had to install system dependencies to the host runners for the CI provider. In method 2, we extract just that step into its own dedicated Dockerfile and then we rely on the CI provider to be able to let us override the runtime environment that executes your steps. In the GitHub Actions ecosystem, it’s normal to have actions that install certain tools, but most tools are unlikely to document how to install the tool itself into a GitHub Actions pipeline. But if you install the tools in your Dockerfile, you can likely just have a RUN command with their suggested install command.

On the downsides, this might be a little complicated still. Most of these CI providers will not let you define the override container to be an image that was built and loaded during the pipeline. That means that the image needs to be already built, and the CI environment needs to have the credentials to pull the image. This is a fairly straight forward one time setup but it can still be rather annoying to do the first time.

Method #3: Run your tests in your docker build

What about a more portable way to build your images? If you run the test command directly in your Dockerfile, then all you need to do to test your code is to build your Dockerfile, and the CI provider that you use doesn’t really matter.

Pros:

• Extremely portable. Every CI provider has the ability to build a Docker image these days

• Running your tests can be completely self contained

• By definition, a successfully built Docker container is a container that passed all of its tests

Cons:

• Building your Docker image can take a lot longer

• Your final Docker image will have all of your test dependencies installed even if you don’t need them to run your application. Bigger images means longer pushes and pulls

• Accessing secrets can be hard

I really enjoy this method for how self contained it is, and the guarantees it creates. If you run your tests in the container image itself, then your ability to move between CI vendors gets easier and easier if you need to, but more importantly, everything is self contained. Your Dockerfile can have all of your operating system level dependencies needed to run your tests, which means that you don’t have to install anything at all on the CI’s host machine. One of my absolute favorite parts about this is the unison of your build environment and your run environment. You know that anything you needed for the tests to run will be available for your application execution. This is really handy for interpreted programming languages. Lastly, every single image that gets built and pushed this way is required to pass your tests. If a test fails, the image build won’t succeed and there’s nothing to push. It’s nice knowing that every single image you have has passed your unit test suite.

On the downside, it can be really inconvenient to build your tests every single time. Remote and containerized development environments are becoming increasingly popular due to how complex our service architectures are becoming. You might be working on multiple services or need your development environments to have massive clusters of compute if you’re a data scientist, so you need a remote enviornment. There are several tools and companies that are trying to enable this workflow like https://skaffold.dev, https://tilt.dev, or paid services like https://okteto.com. These tools will sync your code to remote environments and re-build, re-push, and trigger a re-pull of your containers in a watch mode and reload them in your remote environment. If your organization is advanced enough to have a setup like this, firstly, congratulations. Second, running your tests every time might be a huge pain if your language ecosystem doesn’t have a good way of doing incremental or changed files based test executions. Since you’ve installed all of your test dependencies, the surface area and size of your application runtime container is going to have a lot of extra cruft. For example, in the Python ecosystem, it’s very normal to have tools like psf/black or PyCQA/pylint to be part of your CI steps. But once those are installed, you don’t need them to run your code, but since the exist in the Docker image, they’ll be part of each push and pull of the image. Even worse, sometimes these tools need their own operating system level dependencies that are just going to be laying around and be unused. Lastly, sometimes your CI environments need to access other systems. Passing in secrets or other configuration in general can be really annoying to have to explicitly do every time there’s a new secret or new config file.

Method #4: Run your tests in a multi-stage docker build

Use a Docker multi-stage build for getting rid of the cruft from method #3. Keep separate stages for just your CI, independent of your final image, but keep them in a single Dockerfile for the usability.

Pros:

• Still portable overall

• Smaller image sizes at the end of the day, keep only what you need

• Stages can be built in parallel

Cons:

• More config back on the CI provider

• More complex. Large multi-stage Dockerfiles can be hard to understand

• More redundancy in the Dockerfile itself

• CI commands themselves are not codified in the Dockerfile

• Need to keep better track of which dependencies are dev only vs runtime

I’ve worked on a number of Python and Nodejs based applications that have benefited from an approach like this. You get all of the portability of having your build be self contained, but with a twist. Docker builds have a —target flag which lets you specify to only build up and through a certain stage. In our case, what we do is we isolate all of the dependencies needed for CI in one stage, and then in CI we build to that stage, then run all of our test commands, and then we can build the rest of the image with our normal —target.

Approaches like this try to take a middle ground on the pros and cons we’ve seen up to this point. In a remote development environment, you can do your remote development with an earlier target and without running any of the test commands.

Unfortunately, depending on the programming language you’re using, there might be a lot of redundancy. A lot of package managers will let you specify if you want to install just dev dependencies or prod dependencies. For running your tests, you usually have to install them all, and then in a different stage you have to run the same command but without the dev dependencies. It’s not super common that package managers have a command for uninstalling just the dev dependencies. Another con to this approach is that for things like operating system dependencies, it can be really hard to know what you need for CI steps and what you need for runtime. In package managers like a package.json, there are separate sections for dev dependencies, but this doesn’t always translate easily to the fact that tools like nodejs/node-gyp need Python to be installed to execute, but once artifacts are built, you no longer need to have Python or node-gyp installed at all.

Conclusion

In this post we went over a few different ideas for how you can leverage Docker for doing your CI, on top of having your applications running in a container. In an upcoming newsletter post, I’m going to go over how I made method #3 outrageously fast on my Golang docker builds.

All code samples can also be seen on GitHub by viewing this repo.

I love docker buildx bake. Also, I was really proud of the newsletter post title and the reference to the Dua Lipa song.

I like experimenting with a lot of different ideas, and for the last several years, I've been trying to figure out how to maximize my ability to try new ideas with as little work as possible. In this post, I want to go over what I've settled on these days, which is to use a monolithic git repository, and to leverage Docker's bake functionality. We'll talk about how I structure my repository, how I build the containers, etc. If you'd just like to look through the repo, check it out at abatilo/newsletter-bake-monorepo.

For the backend services, I write web services with Go. I define a global go module at the root of my repository. Go workspaces are a thing as of Go 1.18 but their value is largely when you want to make it easy to work on module changes across repos. For frontend applications, I use create-react-app react applications. I'm not much of a frontend developer, and even though I know there are benefits to some of the newer frameworks, this one is comfortable to me which lets me focus on building. With the react applications, I do leverage npm workspaces. I serve the static assets directly from my Go application, which means that there's only one application container that I need to worry about. That also means that I can serve the frontend and backend on the same domain, which means I get to avoid any kind of CORS configuration. I serve the applications behind CloudFront as my CDN, and so there's still static asset caching, and backend applications get API acceleration, DDoS protection, and more thanks to the CDN. I organize building the containers using a docker-bake file. Bake is a high level build command available to people using docker buildx. In it's simplest definition, you can write a file in the Hashicorp terraform format, and running bake will build all of the targets, and will share common image stages when it can, and will traverse the dependency tree in parallel as much as possible. You can easily define several docker containers, and with a single command, build all of them, very comparable to having a Makefile.

With this high level overview, let's break it down.

One Dockerfile is all it takes, falling in love with bake

Here’s a link to the whole Dockerfile. How’s this work? It’s a multi-stage build where we build the static assets in one stage, build the go binaries in another stage, and put everything together in the last stage.

The frontend stages:

FROM node:18.10.0-alpine as node-modules

WORKDIR /app/web
COPY ./web/package.json ./web/package-lock.json ./
COPY ./web/app1/package.json ./app1/
COPY ./web/app2/package.json ./app2/
RUN npm ci --no-audit

FROM node-modules as node-builder
COPY ./web ./

ARG app
RUN npm run build --workspace ${app}

So we start out with a stage that’s exclusively for installing dependencies. With npm, if you want deterministic builds, you want to use npm ci for your build environments. And with npm workspaces, you have to copy in the package.json for each workspace so that the npm ci can make sure to install everything needed. There are ways to install the dependencies for a single workspace, but since we’ll end up configuring our docker cache for the entirety of this stage, downloading the cache all at once actually ends up being significantly faster than downloading the dependencies for each workspace in parallel. Especially since there’s a lot of shared dependencies between each website. We keep the node-modules itself as an independent stage so that when we cache it, it doesn’t need to worry about the ARG for the app that we want to actually build. That’ll make more sense when we get to looking at the contents of the docker-bake.hcl

The backend stages:

FROM golang:1.19.0 as go-modules

# Install dependencies
WORKDIR /app
COPY ./go.mod ./go.sum ./
RUN go mod download

FROM go-modules as go-builder

COPY ./internal ./internal
COPY ./cmd ./cmd
RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go install -ldflags="-w -s" ./cmd/...

Same as the frontend stages, we have a stage that’s for go dependencies, and we download them all at once and we’ll cache the entire stage. And as opposed to the frontend build, we actually just compile the entirety of the repo. The Go compiler is extremely fast, and in a future newsletter post that I have planned, we’ll talk about how to leverage the Go compiler build cache to make this even faster. Consider subscribing to find out how I do that in a Dockerfile!

Subscribe now

The final stage:

FROM gcr.io/distroless/static-debian11:nonroot
ENV STATIC_ASSETS_PATH /static

ARG app
COPY --from=node-builder /app/web/${app}/build /static
COPY --from=go-builder /go/bin/${app} /usr/local/bin/entrypoint
ENTRYPOINT ["entrypoint"]

Just like I mentioned, we take the static assets for a specific app and the statically linked binary and put those into the same final container.

Writing your docker-bake.hcl

The common config:

variable "GITHUB_SHA" {
  default = "latest"
}

variable "REGISTRY" {
  default = "ghcr.io/abatilo/newsletter-bake-monorepo"
}

group "default" {
  targets = [
    "app1",
    "app2",
  ]
} 

I use the GitHub SHA for my container tags but you can use whatever you want. Writing a variable block in the bake file will automatically be populated by any environment variables that are available when the docker buildx bake command is invoked. In GitHub Actions, GITHUB_SHA is always populated. Setting a default means that I get to build the containers locally without having to remember to set GITHUB_SHA.

The next variable block is for the container registry prefix to push to, without the image name. This one I don’t actually override anywhere but there isn’t a way to set local variables like in Terraform with locals blocks. So I define the registry in one place and we’ll use it for each application’s container target.

group “default” is the default target to build if all you do is run docker buildx bake without any additional arguments. I define all of the images in my project here.

Defining the cache targets:

target "node-modules" {
  target = "node-modules"
  cache-from = ["type=gha,scope=node-modules"]
  cache-to = ["type=gha,mode=max,scope=node-modules"]
}

target "go-modules" {
  target = "go-modules"
  cache-from = ["type=gha,scope=go-modules"]
  cache-to = ["type=gha,mode=max,scope=go-modules"]
}

Bake will build a dependency graph for any shared docker stages that get used by later images, and will walk the dependency graph to build the different targets. This is awesome for the monorepo and how I structured the Dockerfile. It means that I can define the dependency stages as their own images with the cache config, and when I run docker buildx bake, these two targets will be built first and since bake knows that the different apps depend on these targets, we build the dependencies first before them getting used in parallel for each application build. Buildkit has a backend for specifically using the GitHub Actions cache API which is incredibly easy to use. You can read more about this cache backend here.

target "app1" {
  contexts = {
    node-modules = "target:node-modules",
    go-modules = "target:go-modules"
  }
  args = {
    app = "app1"
  }
  tags = [
    "${REGISTRY}/app1:${GITHUB_SHA}",
  ]
  cache-from = ["type=gha,scope=app1"]
  cache-to = ["type=gha,mode=max,scope=app1"]
}

target "app2" {
  contexts = {
    node-modules = "target:node-modules",
    go-modules = "target:go-modules"
  }
  args = {
    app = "app2"
  }
  tags = [
    "${REGISTRY}/app2:${GITHUB_SHA}",
  ]
  cache-from = ["type=gha,scope=app2"]
  cache-to = ["type=gha,mode=max,scope=app2"]
}

Now, we have examples of the actual definitions for a single application’s docker container. Notice that we set the contexts key here which references the dependency targets we just defined. You can basically think of this like the depends_on block if you’ve used Terraform before. The args key let’s us populate the ARG variables in the Dockerfile. This is what is ultimately different between each of the containers. In the future, the buildx team might support using for_each loops like in Terraform but for now each block will have a bit of duplication.

What does CI look like?

There’s an amazing docker/bake-action which makes it insanely easy to build all of your containers in the most optimal way. Since we’ve set the group “default” block in the docker-bake.hcl, config is very minimal. One step in your GitHub Action workflow file will build all of your images and will push all of your cache layers, tag all of your containers, and push all your final images. You’ll still have to do things like checkout the code and don’t forget that you’ll want to use the docker/setup-buildx-action since bake is a buildx feature. There’s one quick gotcha for the actual docker/bake-action. We don’t want to push PR builds and we don’t want to pollute the cache with PR builds.

      - if: github.event_name == 'push'
        uses: docker/bake-action@v2.3.0
        with:
          push: true

      - if: github.event_name == 'pull_request'
        uses: docker/bake-action@v2.3.0
        with:
          push: false
          # Explicitly remove pushing to cache to not pollute the cache with PR
          # specific layers.
          set: |
            *.cache-to=

The way we set cache-to to an empty string is inspired by int128/docker-build-cache-config-action. The wildcard sets the cache-to property for all targets.

Wrapping up

Reminder that a full blown example repo is available at abatilo/newsletter-bake-monorepo. This project structure makes it so insanely easy to add new projects to CI and makes it easy for me to continue to try new ideas. Maintenance cost is very low since it’s a single Dockerfile and the Dockerfile itself is very minimal. These docker builds are optimized for small docker images, parallel builds, and let’s me try new ideas without much effort.

I hope you give docker bake a try. I think it’s a great way to maintain many images in the same repo and keep builds optimal and simple.

I'm a huge fan of open sourcing small utilities, but I never want to release something without automating new versions. As best as I can tell, projects always look more serious and stable if there’s multiple versions that have been published. In this post, I'm going to go over everything that I did to prepare and automate versioning and releasing an open source project. I've used the same base layout for several projects like actions-poetry (261 stars), typed-json-dataclass (79 stars), release-info-action (6 stars), and have even released company internal tools the same way, but today we'll be focused on my most recent utility, github-ratelimit-metrics and the combination of settings that I use for making releases as easy as possible for myself.

There’s three parts to making this work well:

1. GitHub Actions

2. amannn/action-semantic-pull-request

3. semantic-release/semantic-release

We’ll be releasing these projects with semantic versioning (SemVer), which I appreciate as a consumer, because it let’s me easily understand whether or not I should expect an update to impact my usage.

So the first thing that we want to do is make sure that the repositories we want to configure this all on, have squash merging with the following default:

The default is relevant because you might accidentally forget to include a conventional commit header with a commit which will prevent the later automation from knowing how to update the semantic version.

Then configure your repository with a GitHub Action that uses amannn/action-semantic-pull-request. The README has the workflow that I pretty much copy paste into my projects.

This should help ensure that you always keep a conventional commit header in your PRs. Conventional commits are a specification that maps directly to knowing how you should change the numbers that are part of your semantic version. You can prefix a commit with something like feat: which would correspond to a minor version bump, or fix: which would correspond to a patch version bump.

Next, you want to configure semantic-release which is primarily a tool for automatically building and publishing packages to NPM, but with a little bit of configuration, it ends up being an amazing tool for general release management.

semantic-release has a large list of available plugins which will let you control what platform you publish to and what documentation to create.

I place the following content into a file called .releaserc in the root of my repositories:

{
  "branches": ["main"],
  "plugins": [
    "@semantic-release/commit-analyzer",
    "@semantic-release/release-notes-generator",
    "@semantic-release/github"
  ]
}

The default branch for semantic-release is still master, so I switch it to using main. And then I use the commit-analyzer, release-notes-generator, and github plugins.

The commit-analyzer is what will read our conventional commit headers, the release-notes-generator will create a changelog and the github plugin will make releases on GitHub, using the generated changelog for the release description and will comment on related issues and related pull requests. These plugins together can be used for any kind of project. GitHub Actions, docker containers, etc.

One of the reasons that I really appreciate using semantic-release is for the commenting features. When semantic-release is executed, a comment is posted back to the pull request when a PR is included in a release, and if the pull request mentions fixing an issue, the tool will also comment on the issue letting watchers know that the fix has been released.

I love this feature for letting contributors know when their new features are available, or when bugs have been fixed. The contributors can instantly know which version they need to upgrade to to get the changes.

So how does this all get linked together? Let’s look at the release flow for github-ratelimit-metrics.

You can look at the full workflow here but since it’s large, we’ll break it down in this post.

Every workflow release job starts the same

      - uses: actions/checkout@v3

      - name: Generate release
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        run: npx semantic-release

First we check out the repo so that we can get the .releaserc, and then immediately run npx semantic-release. GitHub Actions runners always have an LTS version of nodejs already installed which includes npm and npx. npx is a tool which will download and execute the latest version of an npm package. So executing npx semantic-release will immediately execute semantic-release which will automatically pick up the .releaserc. And we pass in the GITHUB_TOKEN from the workflow which has all of the permissions that it needs by default.

If there’s a new version warranted by the commits since the last time this workflow was ran, then semantic-release will publish that to the GitHub releases. In the case of a Docker based project, now we can fetch that version so that we can tag the container.

      - name: Get Latest Release
        id: latest_version
        uses: abatilo/release-info-action@v1.3.1
        with:
          owner: ${{ github.repository_owner }}
          repo: ${{ github.event.repository.name }}

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v2

      - name: Login to GitHub Container Registry
        uses: docker/login-action@v2
        with:
          registry: ghcr.io
          username: ${{ github.repository_owner }}
          password: ${{ secrets.GITHUB_TOKEN }}

      - name: Prep docker tag
        uses: docker/metadata-action@v3
        id: metadata
        with:
          images: ghcr.io/${{ github.repository }}
          tags: |
            type=raw,value=${{ steps.latest_version.outputs.latest_tag }}

So let’s fetch the latest release version for this repo, and then we’ll pass the output of the latest tag into the docker/metadata-action action.

      - name: Set docker cache tags
        uses: int128/docker-build-cache-config-action@v1
        id: cache
        with:
          image: ghcr.io/${{ github.repository }}/cache

      - uses: docker/build-push-action@v2
        id: build
        with:
          push: ${{ github.event_name == 'push' && github.ref == 'refs/heads/main' }}
          tags: ${{ steps.metadata.outputs.tags }}
          labels: ${{ steps.metadata.outputs.labels }}
          platforms: linux/amd64,linux/arm64
          cache-from: ${{ steps.cache.outputs.cache-from }}
          cache-to: ${{ steps.cache.outputs.cache-to }}

Then we can build our container using the tags and the container will get pushed.

And that’s all it takes! If you’re releasing a GitHub Action, you can skip all the parts about building a docker container. If you’re releasing a cross compiled go binary, you can run something like the goreleaser-action directly after the semantic-release step, and goreleaser will actually edit the release created by semantic-release with all of the compiled binaries. This way, you get all the comment automation and changelog automation from semantic-release and still get all of the binary compilation from goreleaser all on the same release.

At work we’ve been building a lot of GitHub automation for our monorepo. My team is in charge of a lot of infrastructure and tooling, and we’re always trying to find ways to leverage automation to making our product engineers more productive. That being said, we’ve recently run into the problem where the single GitHub App that we’ve been doing most of our automation under, has been hitting its 15,000 requests per hour rate limit. For both security reasons and these rate limit reasons, we’ve started creating multiple GitHub Apps with different responsibilities.

The first time we hit a rate limit, it was extremely disruptive to our eng org. A lot of the automation that run are CI checks on pull requests, and many of them are required to pass before someone can merge in a pull request. Hitting the rate limits left dozens of our engineers unable to merge and ship code, and we didn’t have any proactive monitoring for this use case. I did some looking around and was surprised to find out that there actually wasn’t really much available for this as a ready to use solution.

Another option that I came across would be to create a GitHub Action workflow where you print out your current rate limit status. You could have a workflow like below but it would be really hard to alert on that.

The next post that I came across outlines a python script that you can run to expose the metrics as prometheus metrics, but it only gives you snippets. There’s no fully working project that you can use. But this project is pretty close to something we could have used.

Moreover, there’s no way to get the rate limits on behalf of another entity. So if you want to get the remaining number of requests for a GitHub App, you need to authenticate as that app and fetch an installation token that’s specific to that app’s installation to your repo.

Fortunately, you can get remaining requests either from parsing HTTP headers but there’s also a dedicated rate limit endpoint which is what we used in the example workflow. Having a dedicated endpoint is nice since that means we don’t have to have every single use case of an app’s credentials independently report on its remaining headers.

In comes my latest project github-ratelimit-metrics. A web server that exports remaining limits as a prometheus metric, very similar to the python scripts mentioned in the blog post I linked above. With a configurable interval, it’ll check rate limits and update the prometheus gauges with the latest values. Along with the project itself, I’ve put together a helm chart for the project to make it easy to deploy in Kubernetes environments, but since github-ratelimit-metrics is actually a go static binary, it’s both very easy to run just as a binary, and also is already built and published for both linux/amd64 and linux/arm64 architecture compatible docker containers.