The month the free minutes ran out

This is the fourth of four posts. The first built a DevSecOps pipeline for free on a private GitHub repo. The second was three weeks of living with it. The third found out how little the scanner was reaching and fixed it with an API-definition scan. This one is about the bill for all that, which arrived on 28 September in a form I hadn’t budgeted for.

The day GitHub stopped starting jobs

Release v0.29.0 went out on 28 September and its image job never started, which is a different thing from failing. The annotation on the run, verbatim: “The job was not started because recent account payments have failed or your spending limit needs to be increased.” The release had already published, so its deploy waited ten minutes for an image that was never coming and then failed. The bot stayed on the previous version: the deploy had failed before it touched the server.

The cause was the security workflow. Two ZAP scans on every PR, and the same two again on every push to main, because the workflow ran on both. One of those runs had also waited ten minutes for a ZAP that died on startup. By the 28th that had used up the 2,000 free Actions minutes a private repo gets a month: GitHub’s meter showed 2,000 of 2,000, and $12.09 of gross usage for this repo in September, all of it covered by the allowance right up to the point where there was none left. The first post’s thesis was that DAST was cheap, and it was, in engineering. In minutes it was the most expensive thing in the pipeline.

Two smaller gate stories from the day before, both true to the series’ theme. Semgrep had been flagging direct-response-write on a Fastify route that did reply.send(pantryToDTO(...)), at WARNING, so never blocking, and it had been in every report since at least 23 September. I only noticed when I looked at the job summary instead of the exit code, which is the third post’s lesson arriving late. It was a false positive: the value is an object Fastify serialises as JSON. I fixed it by returning the DTO from the handler, like the other routes do, so the pattern disappears instead of being silenced with a nosemgrep comment, and the next scan reported 0 findings. And the ZAP API scan failed once, in 12 runs, with “Address already in use”: ZAP’s script picks a random free port, then starts the daemon on it, and lost the race. The daemon died and the script waited ten minutes for it. Each of the three ZAP scans now gets a fixed port.

A box of my own

I bought a second VPS, 4 GB of RAM and 2 cores, no commitment. Separate from the bot’s on purpose: a runner executes other people’s code, every Dependabot bump and every install script, and nothing on it should be able to reach production. The one thing that can is the deploy key, which lives in GitHub and exists on the box only during a deploy job. All four workflows moved to runs-on: [self-hosted, linux, x64]. The setup is a script in the repo, run after the hardening script the bot’s own box already used. Every lesson below cost a red run.

The one I’d want someone to have told me: the runner’s work folder has to be /home/runner/work, GitHub’s own layout. actions/cache stores paths outside the workspace relative to the workspace, so a cache saved by a GitHub-hosted runner contains ../../../.local/bin/trivy-bin, and on a box whose workspace was /opt/actions-runner/_work/... it unpacked into /opt/actions-runner/.local. The log said “Cache restored successfully” and then “trivy: command not found”, in the image job and in the IaC job, and both lines were telling the truth 😅

The second: one runner is a queue. A single change queues seven jobs (CI and security for the PR, then CI, security, release and deploy for main), and they run one at a time. The v0.30.0 deploy, the next release, took the only runner and sat waiting for an image that only the CI job queued behind it could build. So I added runners.

The third: three runners on two cores was too many. Load average 10, and the Postgres adapter’s tests, which boot Postgres compiled to WASM in a beforeEach, blew vitest’s default 10-second hook timeout on main and on PRs. Now it’s one runner per core, two, and hookTimeout is 60 seconds while testTimeout stays at the default: a slow test is a finding, a slow boot is the machine. Peak memory during a scan was about 1.6 GB. Memory was never the problem; CPU was.

The smaller ones:

• The runner-registration token copied from the web page answered 404 twice. Generating it with gh api -X POST and piping it over SSH worked first time.
• gh isn’t on a bare box, so the first deploy succeeded and then failed at the very last step, attaching the SBOM, with exit code 127.
• Re-running the setup script stopped and restarted every runner service and cancelled the three jobs they were running: “The operation was canceled”, in steps that cannot fail on their own. I read it as an OOM until the journal showed systemd stopping the services one after another. The script now restarts a runner only when its config changed, and only once it’s idle.
• The DAST job boots the app on port 3000 and ZAP on fixed ports of the host, so several runners on one box got a repo-wide concurrency group: one DAST at a time.
• Two more the day after. needrestart, which restarts services after package upgrades, was restarting the runners mid-job. And the box was filling up with stopped containers until a daily prune went in.

What the pipeline does now

Security runs on every PR and weekly on main, Mondays at 04:17, with the active ZAP full scan. It no longer runs on push to main: main only moves by squash-merging a PR whose merge commit the scans already passed, so that run scanned the same bytes twice. The ZAP API scan, the active scan of every OpenAPI route from the third post, is weekly-only too: it had been report-only since 18 September and had gated nothing, and it was four of a PR’s eight minutes.

CI on a PR is two jobs in parallel: verify (audit, typecheck, test, build, shellcheck) and image (docker build plus Trivy, HIGH and CRITICAL blocking), without pushing anything. CI on push to main is verify only.

The registry holds release images only. The deploy workflow, on release: published, checks out the tag, builds the image, scans it with Trivy, pushes it as vX.Y.Z, deploys over SSH (a forced-command key, a health check with rollback) and attaches a CycloneDX SBOM to the release. Before, every merge pushed a sha-<commit> image nobody deployed and the release re-tagged it, “so the bytes tested are the bytes shipped”. That guarantee is weaker now, on purpose: same commit, same lockfile, same base image tag, but the Debian packages are those of the release day, not the merge day, which in practice is what you want from a patch-at-build image. For one day there was a design in between, where the deploy also fired when CI finished and whichever arrived second shipped; it worked, two releases went out that way, and it was gone the next day, when the deploy started building its own image. Every release since has shipped that way.

Two things that aren’t decisions but saved time: builds use buildx’s docker driver, so the daemon’s persistent layer cache serves the next run instead of the previous type=gha cache doing a round trip to GitHub on every build; and pnpm comes from packageManager via corepack, kept on the box, instead of pnpm/action-setup downloading it on every run.

What it costs now

Measured on the box, two cores, per job. Before the changes above: DAST 468 s (among them the API scan 241, the baseline 128, install and build about 40), image 130 s (93 of them the build), verify 109 s (tests 38, downloading pnpm 19), SAST 36, IaC 38, Secrets 16, SCA 13. A PR took about 8 minutes of wall-clock, bounded by DAST. After: DAST 111 s, image 83 s, verify 103 s, and a PR about 4.5 minutes with two runners.

	GitHub-hosted (second post)	Self-hosted, before	Self-hosted, after
DAST	2m04s	468 s	111 s
image	in the 1m46s CI	130 s	83 s
verify	40–46 s	109 s	103 s
PR, wall-clock	2m10s	~8 min	~4.5 min

Corepack will save the 19-second pnpm download from the second run on; the run in the table was still downloading it, which is why verify only dropped six seconds. The docker driver explains the 47 that came off image. The box is slower per job than GitHub’s runners and unlimited in minutes, which is the trade I was actually making when I wrote “free” in the first post without knowing it.

Things I decided not to do, with the reasons written down so I can be wrong about them later. A 4-core box, at double the price: the critical path is a single DAST job that doesn’t parallelise, so four cores save about three minutes only when several PRs are in flight. I’ll look again at the end of October, together with the annual plan. And a secrets manager for the bot’s .env on a single VPS: Docker keeps container environment in plaintext in its own state anyway, and the process needs the secrets in memory, so on one box Vault or sops would move the secret, not the risk. A real one, with secrets fetched to a tmpfs and read through *_FILE, is the next piece of work.

What I’d add to the three posts before this: count minutes as well as seconds. A scan that is free to build can still empty a monthly quota, and the number that told me wasn’t in any of my tables; it was a red bar on GitHub’s billing page.