A CI gate that is sometimes red and sometimes green looks, at a glance, like a flaky test. On the main video-generation service, the gate spent roughly a month doing something different: it kept getting refused by GitHub itself, for reasons that had nothing to do with the code under review, in a pattern specific enough that the team started numbering the incidents. By the time it was over, they'd counted seven flips — and the honest accounting of what that month actually cost turned out to be as valuable as the fix. Alongside it, a separate but related cluster of bugs was quietly letting single paid renders run far longer, and far more expensively, than anyone intended.
A flaky gate you can ignore case by case. A gate that is unsatisfiable by construction will train everyone who touches it to stop trusting green, which is worse than training them to ignore red.
The shape that isn't a test failure
The first thing worth understanding is that most of these "flips" weren't CI failing — they were CI never starting. PR #1171, the fifth recorded instance, captured the exact signature: a Merge Gate job reporting conclusion=failure with steps=0 in three seconds flat, Build and Unit Tests both silently skipping, and an annotation reading "the job was not started because recent account payments have failed or your spending limit needs to be increased." That's a billing failure on the CI provider's account, surfacing as a red check indistinguishable at a glance from a real test failure — unless you know to look at the step count. A job with zero steps didn't run long enough to fail anything; it was refused before it started.
The team's own fix for telling the two apart wasn't a script — it was a paragraph in CLAUDE.md instructing every future reader to distinguish the shapes: a real failure has steps and a duration; an account block has neither. That paragraph needed updating five separate times as the state kept recurring, and each update recorded the "cleanest specimen yet" so the next person wouldn't have to re-derive the diagnosis from scratch.
A gate that was unsatisfiable, not merely unlucky
The sixth flip, PR #1175, is the one that mattered most, because it found CI finally executing for the first time since September 1st — and discovered two real conditions sitting underneath all the noise, neither one a regression in the code being tested. The first was a build step running under three different names across two workflows, all hitting the exact same out-of-memory ceiling; patching one name by itself had left the other two silently broken. The second was sharper: a test suite that drove real data from a directory where exactly one file was actually tracked in git. On a CI runner, that directory is empty by construction, so the test's own guard against a vacuous pass had been failing — correctly — since an earlier PR, number 1011, with nobody connecting the dots that the test had never once passed in CI.
The fix doesn't relax the test. It separates "the corpus directory is absent" (skip loudly, and say explicitly that a skip is not evidence of anything) from "the corpus directory is present but too short" (fail, exactly as before). That distinction — a permanently red guard teaches people to skip it, but a guard that goes vacuously green teaches them to trust a lie — is the same discipline a dead-man's-switch needs anywhere: silence and success have to be distinguishable states, or the gate stops being a gate.
| Flip | What actually happened |
|---|---|
| Fifth | Account-block signature finally pinned down precisely, by shape |
| Sixth | First green run since Sept 1 — found the real OOM and vacuous-corpus bugs underneath |
| Seventh | A partial block — only the final summary job refused, four real jobs upstream had already passed |
Getting the diagnosis wrong, in writing, on purpose
The seventh flip, PR #1221, is the most interesting entry in this whole saga because the PR description documents its author's own mistaken first read. The block landed mid-run, so four real jobs — lint, build, unit tests, and the merge gate itself — had already started and completed successfully before a fifth job, the PR summary, was refused with the same billing annotation. The author's first instinct was: the other jobs ran for fifteen minutes, so the account can't actually be blocked. That reasoning is wrong whenever a block arrives partway through a run rather than at the start, and the PR corrects it in the same document that contains the mistake, specifically so the next person doesn't re-derive the same wrong conclusion. A partial block, it turns out, is also the cheapest version of this failure to live with: the substantive gate genuinely ran and its green result is real evidence — only the final summary comment is missing, which calls for naming the refused job rather than writing off a mostly-green run as entirely red.
PR #1176 and #1224 close the loop with the part of this story that's easiest to skip: a direct accounting of what a month of unreliable CI actually cost. Three separate breakages — the memory ceiling, the vacuous test corpus, and a duplicate YAML key that silently invalidated an entire workflow file in a way that never showed up in the normal gh pr checks output — had all been accumulating underneath the account-block noise, invisible precisely because the account block was the more obvious, more dramatic failure mode sitting on top of them. #1224 goes further and tallies real numbers: 616 merged PRs since the last clean baseline, cross-checked against the actual dev/main branch divergence (zero, despite an earlier assumption that dev was lagging) and a re-priced cost line in the roadmap docs that had been wrong by roughly 20% because nobody had gone back to update a constant after a vendor's real billed rate was measured.
The other half: runaway cost from processes that never die
A parallel cluster of fixes this cycle tackled a different flavor of the same root problem — a safety mechanism that can't fire because the failure it's guarding against doesn't look like a failure. PR #1109 found that an Anthropic streaming call had no bound on total output: max_tokens is where the model stops, but the code was draining the entire stream via finalMessage() before ever checking the stop reason. Measured directly on two real jobs: a healthy response used 6,254 output tokens; a runaway one, given the identical prompt, generated 64,000 tokens over eight minutes before being caught as truncated and thrown away — all of it already billed. The fix bounds total output tokens directly and aborts the stream early, stopping the runaway "at under a third of the price" instead of waiting for it to finish failing.
PR #1117 found a related but distinct bug: a vision call that stalled completely — zero output, zero error — for 75 minutes, holding a render open that had already cost $37.87 in rendered clips. It never threw, so every .catch written around it was inert — a hang isn't an exception, so exception-handling code can't see it. The fix bounds idle time rather than total time, specifically so a legitimately long generation that's still producing text is never cut short, while true silence trips a watchdog. PR #1118 found the same failure mode one layer down, in an ffmpeg child process: a stitch that could hang forever is, in the PR's own framing, "an ad that never ships" — not a metaphor, a literal description of the failure. And PR #1119 is the fix that made the other two trustworthy: the kill mechanism for a stuck process was firing silently, leaving no trace of what it had killed or why — which meant diagnosing a slow render meant re-discovering the same root cause from scratch every time. Three separate bugs were investigated and ruled out by actual measurement before the real one, a 30-second-per-panel contact-sheet render wedging ffmpeg past roughly 30 panels, was found — because the first local repro used a cheap, downscaled test image that made an expensive filter graph look fast.
The pattern: a safeguard only works if its trigger condition can actually occur
Every failure in this cycle — the CI account-block, the vacuous test corpus, the inert .catch blocks, the silent kill — shares the same shape. Each one had a safety mechanism already in place, written by someone who had reasoned correctly about the failure they expected. What none of them accounted for was a failure mode where the triggering condition itself never arrives: a job that never starts isn't a failed test, a hang isn't a thrown error, a silent kill isn't a kill anyone can learn from. A safeguard is only as good as its ability to detect that it should have fired at all — and the fix, every time in this cycle, wasn't a smarter retry or a bigger timeout. It was making the absence of a signal into a signal of its own.