How we count GitHub API requests

If you talk too much, GitHub will shut you up

Octocat holding too many books, generated by DALLE-2

At work we’ve been building a lot of GitHub automation for our monorepo. My team is in charge of a lot of infrastructure and tooling, and we’re always trying to find ways to leverage automation to making our product engineers more productive. That being said, we’ve recently run into the problem where the single GitHub App that we’ve been doing most of our automation under, has been hitting its 15,000 requests per hour rate limit. For both security reasons and these rate limit reasons, we’ve started creating multiple GitHub Apps with different responsibilities.

The first time we hit a rate limit, it was extremely disruptive to our eng org. A lot of the automation that run are CI checks on pull requests, and many of them are required to pass before someone can merge in a pull request. Hitting the rate limits left dozens of our engineers unable to merge and ship code, and we didn’t have any proactive monitoring for this use case. I did some looking around and was surprised to find out that there actually wasn’t really much available for this as a ready to use solution.

Another option that I came across would be to create a GitHub Action workflow where you print out your current rate limit status. You could have a workflow like below but it would be really hard to alert on that.

This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters

Show hidden characters

	name: Print rate limits
	on:
	push:
	jobs:
	rate-limit:
	runs-on: ubuntu-latest
	steps:
	- name: Generate token
	id: generate_token
	uses: tibdex/github-app-token@v1
	with:
	app_id: ${{ secrets.APP_ID }}
	private_key: ${{ secrets.PRIVATE_KEY }}
	- env:
	GITHUB_TOKEN: ${{ steps.generate_token.outputs.token }}
	run: |
	curl \
	-H "Accept: application/vnd.github+json" \
	-H "Authorization: Bearer $GITHUB_TOKEN" \
	https://api.github.com/rate_limit | jq '.'

view raw ratelimit.yml hosted with ❤ by GitHub

The next post that I came across outlines a python script that you can run to expose the metrics as prometheus metrics, but it only gives you snippets. There’s no fully working project that you can use. But this project is pretty close to something we could have used.

Moreover, there’s no way to get the rate limits on behalf of another entity. So if you want to get the remaining number of requests for a GitHub App, you need to authenticate as that app and fetch an installation token that’s specific to that app’s installation to your repo.

Fortunately, you can get remaining requests either from parsing HTTP headers but there’s also a dedicated rate limit endpoint which is what we used in the example workflow. Having a dedicated endpoint is nice since that means we don’t have to have every single use case of an app’s credentials independently report on its remaining headers.

In comes my latest project github-ratelimit-metrics. A web server that exports remaining limits as a prometheus metric, very similar to the python scripts mentioned in the blog post I linked above. With a configurable interval, it’ll check rate limits and update the prometheus gauges with the latest values. Along with the project itself, I’ve put together a helm chart for the project to make it easy to deploy in Kubernetes environments, but since github-ratelimit-metrics is actually a go static binary, it’s both very easy to run just as a binary, and also is already built and published for both linux/amd64 and linux/arm64 architecture compatible docker containers.