Three weeks with the gates: what they blocked, what they got wrong, and what they made me fix

The part almost nobody writes up is what happens after the pipeline goes green. So here goes.

If you haven’t read the first post, the short version: a free DevSecOps pipeline on a private GitHub repo, for a Telegram bot that keeps track of the food in my kitchen, with seven scanners pointed at a throwaway copy of the app that boots inside the runner with no database, no API key and no bot token. From day one, three things blocked a merge (a committed secret, a Semgrep ERROR, a HIGH/CRITICAL misconfiguration) and ZAP only reported, until I knew how much noise it made. The first run found a shell injection in my own deploy workflow, and Trivy blocked the first merge.

This is what the next three weeks looked like.

Three weeks later

The ZAP gate went on, with the number in hand. Fourteen runs and eleven days later the volume was five Lows, stable across baseline and active scans. So fail_action: true, plus a file in the repo, .zap/rules.tsv, where every alert ZAP has ever raised gets a written answer: fixed, or ignored with the reason next to it. First run as a real gate: 0 FAIL, 0 WARN, 63 passive rules passed. That counts rules, though, not routes; how far the spider actually gets is a different and much smaller number, and the third post is about it. Starting in report-only mode wasn’t me being soft. It meant that when I flipped the switch I had a number to point at, rather than a feeling.

There’s a price, though. With the gate on and only the known alerts answered, anything new turns a PR red: a route I add, but also a rule ZAP adds, because the scan runs zaproxy:stable and refreshes its add-ons from ZAP’s marketplace at the start of every run. So I’ve built a gate that can cry wolf on someone else’s schedule, and I’m fine with that, because I’d rather learn about a new ZAP rule from a red check on a PR that touched nothing than not learn about it at all. When it happens, the answer is one more line in rules.tsv.

Four of those five Lows were real, and got fixed rather than ignored. X-Content-Type-Options was set on the page but missing from the JSON routes, where Caddy had been covering for the app. The other three were the cross-origin headers: Cross-Origin-Resource-Policy was set nowhere, and COOP and COEP were missing from the page, where they’re safe because the CSP is default-src 'none' and the page opens no popups. What the scan had actually measured was the gap between what the app does on its own and what Caddy adds in front of it, and that’s a gap you can’t see by scanning production, because there the proxy is part of the response.

One thing I only learned by doing it: those three headers are three alerts under a single ZAP rule id, 90004, and rules.tsv works by id, so you can’t ignore two and fix one.

Permissions-Policy was the fifth, and it’s deferred. ZAP only checks that the header exists, so camera=(self), microphone=() would satisfy it today. But photographing the printed date on a package is a planned feature, and I’d rather write that policy once, next to the code that needs the camera, than write a placeholder now and forget it’s a placeholder.

HSTS wasn’t on ZAP’s list, and couldn’t have been. The scan talks plain HTTP to 127.0.0.1, and ZAP only reports a missing HSTS header on HTTPS responses. I’d been counting it as one of the five until I reread the report for this post. It got set anyway, in Caddy, where TLS is terminated, with a matcher excluding localhost: a browser remembers HSTS for the whole max-age, and one careless line would force https on every other local project sharing that hostname for a year.

Trivy blocked main a second time, and this one was the operating system. Two HIGH in libpcre2-8-0 at 10.42-1, fixed in 10.42-1+deb12u1 in Debian bookworm. Rebuilding didn’t help: node:22-slim had last been published on 25 August and didn’t carry the patch. Debian ships security fixes days before Docker Hub rebuilds the base image around them, and that happens with every CVE.

Rather than a .trivyignore, the fix was to make the image patch itself:

ARG APT_SNAPSHOT
RUN apt-get update && apt-get upgrade -y \
    && rm -rf /var/lib/apt/lists/* \
    && echo "$APT_SNAPSHOT" > /etc/apt-snapshot

Docker’s own best practices say not to apt-get upgrade in a Dockerfile, and hadolint no longer enforces that: the rule stopped flagging it in 2021 and is gone from the version the action pins. For this image, patching during the build is the practical fix, and the gate is what verifies it worked.

APT_SNAPSHOT is just the date, and the name promises more than it delivers. It isn’t a pin against snapshot.debian.org; it’s a cache-buster. Without it the layer cache freezes, and last week’s apt-get upgrade patches nothing. It changes once a day. Both builds, the scanned one and the pushed one, have to get the same value.

What this trades away is reproducibility. The same Dockerfile on the same day can produce two different images if Debian publishes at noon. I took freshness over reproducibility, for an image that gets rebuilt on every merge anyway, and I can live with that.

Dependabot proposed node:26-slim on day one. It failed: Node stopped bundling corepack in 25, and corepack is how the Dockerfile gets pnpm. Closed, with the reason written down, and not needed anyway: with npm gone and Debian patches applied, node:22 is clean. I keep base-image majors out of Dependabot’s groups on purpose, so they arrive alone and get read on their own.

The three MEDIUMs my gate never showed me

Before publishing the first post, one question came up about my own paragraph on the window between a compromise and its advisory, the one pnpm audit can’t see: where was the fence? Meaning some delay between a version being published and it being allowed in, long enough for a bad release to get noticed and pulled.

I went looking, and what I found was worse than “not there”. It was there twice, and I hadn’t put it there either time. 🙈

Since 14 July 2026, Dependabot waits three days before proposing a version update, by default, with nothing configured. And pnpm 11 ships minimumReleaseAge at one day by default. That “Lockfile passes supply-chain policies” line I’d been scrolling past on every install was a fence I didn’t know I had. The ecosystem had put two fences around the window in the months before I built the pipeline, and I’d written a whole paragraph about the window without noticing either.

The pipeline had tried to tell me, on its very first run. Semgrep had flagged, at MEDIUM, that the Dependabot config had no explicit cooldown: three findings, one per ecosystem. I never saw them.

My gate script knew three severities, ERROR, WARNING and INFO, and Semgrep’s newer rules speak a different scale: CRITICAL, HIGH, MEDIUM, LOW. Anything on that scale fell through both the blocking check and the summary table. The run I read as “two findings” had five. It was the second bug in the same gate script, and it turned up the same way as the first one, with real data hitting it. The check now takes a set (ERROR, CRITICAL, HIGH), and a severity the script doesn’t recognise lands at the bottom of the table instead of vanishing. I went back through every Semgrep report since that first run, and nothing HIGH or CRITICAL had ever fired, so nothing had actually slipped past, which was luck rather than design.

And Semgrep was right even with the defaults in place. A fence you’ve set yourself survives the defaults changing, and it can be longer than the default. Those three MEDIUMs were pointing at the fence I should have set myself.

Four things changed in one PR.

A seven-day cooldown on Dependabot, for all three ecosystems, written down rather than inherited. Security updates bypass the cooldown by design, which makes sense: the delay is meant for new versions, not for fixes to problems everyone already knows about.

pnpm’s minimumReleaseAge, raised from its one-day default to the same seven. What makes it stronger than the cooldown is where it runs: at every install, frozen lockfile included. A lockfile that bypassed the policy on someone’s laptop fails in CI. The first push of that change went red on five packages already in the lockfile, all published in the previous week. They went in as named, dated exceptions, which is the same shape a security fix younger than seven days will take. Four days later the last of them had aged out, and the exceptions went with it.

gitleaks now scans full history on every run. It had been scanning the working tree on pull requests, so a secret added in one commit and removed in the next left no trace and passed; only the weekly history scan would have caught it, after it was on main. That means fetch-depth: 0 on every checkout, which at a couple of hundred commits is free (the scan takes under a second) and at five thousand would be a decision to revisit.

And pnpm audit got the treatment I’d already given Semgrep. With the registry unreachable, it prints {"error": {"message": "fetch failed"}} and exits 1, the same code as “advisories found”, so a registry blip would have read as “vulnerable”. It now exits 2 with no report, 1 with advisories, 0 when clean. The build fails either way; what changes is that you know whether you’re looking at a vulnerability or at the network before you’ve opened a log.

None of these added a job. The CI cost is the same as week one.

The step that was green and had never worked

The image job ends by attaching build provenance. I’d wired it with continue-on-error, having read it wasn’t available on every plan. Then I went to check whether it had ever actually worked.

The job summary said success. That’s what continue-on-error does: it lets the workflow carry on and reports the step as green. You have to open the raw log to find outcome=failure.

It had never worked, not once, from the very first run:

Error: Failed to persist attestation: Feature not available for
user-owned private repositories. To enable this feature, please
make this repository public.

The repo’s attestations API returns 404. There is nothing stored, and there never was.

The uncomfortable part isn’t the plan limit. My continue-on-error did exactly what I asked of it, stop an unfixable failure from blocking merges, and it also painted a step green that had never once worked. That’s the same failure mode as the Semgrep exit code in the first post: something incapable of doing its job, reporting as though it did. I designed for it in one place and walked straight into it in another. And the job still holds id-token: write and attestations: write for that step, two permissions granted for nothing.

The step stays, continue-on-error and all, but the comment above it now says why: it can’t succeed here, and the day this repo goes public (or lands in an org on Enterprise Cloud) it starts working with the wiring already in place. A gate that can’t work and reports as though it can is worse than no gate, unless you’ve written down next to it why it’s there.

What it costs

	Before	After
CI on a pull request	40–48s	40–46s (1m46s with the image build and scan)
Security workflow, in parallel	—	2m10s
Weekly, with the active full scan	—	3m14s

The five security jobs run concurrently: secrets 6s, IaC 17s, SCA 18s, SAST 27s, DAST 2m04s. They add up to about three minutes, but wall-clock is set by DAST alone, the only one that installs, builds and boots the app. Feedback on a pull request goes from roughly 45 seconds to 2m10s. If that’s too much for you, the lever is obvious: move DAST out of pull requests and into the weekly run. These are week-one numbers; the DAST one moves in the third post.

The irony at the end

Nearly every GitHub-platform limitation across these two posts (no CodeQL, no native secret scanning, no upload-sarif, no branch protection, no attestations) has the same one-click fix: make the repo public.

GitHub isn’t even subtle about it. Its own error message ends with to enable this feature, please make this repository public.

The price is publishing my deploy scripts, my Caddyfile and the exact shape of a server that is running right now. Those eight hundred lines of bash from the opening of the first post, the ones nothing had been looking at, would be the first thing anyone looked at. In exchange I’d get, for free, the analysis I’m currently replacing with containers, the merge barrier I don’t have, and an attestation step that has been failing silently since the day I wrote it.

I honestly don’t know which way that falls. And that the answer isn’t obvious is more or less everything worth knowing about doing this on a personal project. 😄

What I’d keep from these weeks, if I had to start over: put every new gate in report-only mode first, and only switch it on once there’s a number to point at. Read the actual report, not the summary, because the summary is where both of my bugs hid. And don’t confuse a step that can’t fail with a step that works.

There’s a third post, and it’s the uncomfortable one: what all of this doesn’t cover, and how little of the app the scanner was actually reaching.

Update, 29 September 2026

The cost table above is history. Those were GitHub-hosted numbers, and on the 28th of the month GitHub stopped starting jobs, with 2,000 of the 2,000 free Actions minutes used up, so the pipeline moved to a self-hosted 2-core box. There, a PR takes about 4.5 minutes wall-clock with two runners (DAST 111 s, image 83 s, verify 103 s): slower per job, unlimited in minutes. The security workflow also no longer runs on push to main, only on PRs and weekly, because main only moves by squash-merging a PR the scans already passed, and that run scanned the same bytes twice. The provenance step moved with the push into the deploy workflow, continue-on-error and all. Details in the fourth post.