Self-hosted GitHub Actions runners with AWS and RunsOn

GitHub Actions is a great tool to run workflows and jobs for your CI/CD system. However for large test suites I was frustrated by the slow build times, and the bill quickly grew out of control as the team expanded.

So I went ahead and created a new tool (RunsOn) that automatically spawns self-hosted runners for GitHub Actions jobs, using EC2 instances. Since CI jobs are by nature short-lived (most of them <30 min), we can use spot instance pricing offered by AWS to greatly reduce the per-minute cost of the instances (~70% less on average).

I was also surprised to see that the runners provided by GitHub have a low CPU performance, so having access to the full AWS inventory means we can get at least 30% better performance by using a more recent CPU (even 50% better performance for arm runners!).

Another benefit of using self-hosted runners with AWS is that we get access to instances with much greater CPU count, which means we can finally make use of parallelisation in our test suites, and stop wasting engineering time splitting our test suites to make use of multiple (CPU-limited) runners.

Imagine how much faster your builds would go with runners powered by 4/8/16/32/64/128 or even 256 CPUs! And you even get access to the whole range of GPUs offered by EC2 if you need them.

RunsOn comes as a software stack that you install in your own AWS account. It will spawn EC2 instances in place of official GitHub runners. It is a drop-in replacement for official or third-party GitHub Action runners, and can do its job thanks to a few fully managed AWS services:

RunsOn supports multi-AZ deployments across 3 availability zones, so your runner instances are distributed across multiple availability zones. This ensures high availability and redundancy, minimizing the risk of downtime and ensuring that your CI/CD pipelines can continue to run smoothly even in the event of an AZ failure.

This also increases the chance of finding spot instances at the best price possible, as the instances are not all in the same AZ.

Instances are ephemeral, and are only started when there is work to do, and terminated when the work is done. So there is no idle cost, and no need to manage an extended lifecycle for the instances.

The only limit to the number of instances you can run in parallel is the number of EC2 instances (spot and on-demand) you can launch in your AWS account. This can be adjusted at any time by requesting a quota increase from AWS. If spotinstances are not available, then on-demand instances will automatically be used instead.

Note that the spot and on-demand instances have their own quotas. The spot quota is the most important to increase if you have a large number of jobs to run.

RunsOn is in production at tens of companies, and has been battle-tested in production since January 2024. It now routinely handles more than 100k jobs/day across all users.