A vendor error called VendorQueuedTooLong carried a comment that read, in effect: "nothing was rendered, so resubmitting is free." The reframing tool's render pipeline believed that comment and acted on it every time a generation sat in queue too long. Both halves of the claim were false, and they cost real money. The vendor pre-charges at submission time and only refunds a generation that reaches a documented failed state — and the "abandoned" generations weren't failing. Twelve for twelve of the ones this exact error killed in production had actually reached completed at the vendor, sitting there, paid for, with nobody ever coming back to collect them.
That discovery is where a 27-PR arc started: build a background sweep that finds generations the vendor has already billed for and reclaims them instead of quietly resubmitting and paying twice. What makes this cluster worth a post of its own isn't the sweep's existence — it's that the sweep itself turned out to need the same scrutiny as the pipeline it was auditing, right down to a defect that made it structurally blind to its own primary use case.
A cost-reconciliation process is still software. It can have the exact bug class it exists to catch.
Measuring the vendor instead of trusting the error message
Across the account's whole history — 1,020 generations between May and September — 1,005 completed normally and only six ever reached a true failed state. The remaining nine had sat in a queued state for 122 to 146 days with no assets attached: the vendor, it turned out, holds a generation indefinitely rather than ever failing it outright, which meant any sweep built on "wait for failed" needed a give-up condition of its own. PR #16 recovered the twelve stranded result URLs directly and, just as importantly, stopped creating new orphans: the queue-timeout error now parks a run in an awaiting_vendor state instead of marking it terminally failed, so a boot scan or sweep still has something alive to reconsider later.
PR #17 is a small, disciplined piece of measurement worth calling out on its own. The render-budget logic depended on observing a specific vendor transition state — dreaming — and nothing had ever confirmed that state was actually observable at the poll interval being used. A history tally couldn't settle it either, since every past generation has already resolved one way or another; a snapshot reads zero dreaming states whether or not the state is really samplable. The only way to know was to spend one real, paid 540p probe and watch the transitions land in real time. It confirmed the state is entered within about five seconds and held for roughly nineteen — comfortably inside the poll window — which is the only reason the rest of the budget logic downstream could be trusted at all.
The sweep that couldn't see the case it was built for
PR #21 is the center of this whole story. A high-effort review of the accumulated sweep code turned up fourteen findings, and the first one undid the premise of everything before it: the sweep could never actually observe a parked run. The in-memory job registry's "live" check returned every non-terminal job, including parked ones — so the very container that had parked a run kept heartbeating it forever, which meant its lease never went stale, which meant both of the sweep's own filters excluded it: the process-local id check, and the stale-heartbeat condition in its query. The one container that could have collected the parked generation was, by construction, the one container that never would. That is precisely the failure mode parking was built to solve, and the sweep had been quietly unable to touch it the whole time.
The same review surfaced a cluster of smaller but equally real defects, each pinned down with a test that fails the moment the fix is reverted:
| Finding | Consequence |
|---|---|
| A hand-back path skipped the resume-attempt cap | the sweep itself could push a run past its maximum resume attempts |
| A give-up wrote nothing to the row | every later cycle re-probed the vendor for a case already settled |
| A write's return value was discarded | a silently dropped write reported as a successful collection |
| The give-up clock read the row's own update time | candidates read as minutes old when they were really 11–19 days old |
| A 404 was treated as a vendor failure | indistinguishable from what a rotated API key also returns |
| Status vocabulary duplicated in three places | a second and third copy could silently drift from the first |
None of these are exotic bugs. They're the ordinary failure modes of any stateful background job — and they existed in the code written specifically to catch ordinary failure modes in the pipeline next door.
A fallback ladder that was buying the same seconds twice
PR #22 found a second, more direct instance of the same root cause. The fallback ladder that escalates across vendor brokers caught every subclass of a generic vendor-error type and advanced to the next broker — including the two states, queue-timeout and stall, that the system's own error documentation explicitly flagged as already billed. The ladder's entire justification had been written as "one queue starving is the whole problem this exists for" — but a starved generation is, by the vendor's own pricing model, one that's already been accepted and paid for. Advancing past it didn't recover from a starved queue; it bought the same seconds a second time on a different broker. The fix narrows which error types are allowed to advance the ladder to ones that provably bought nothing — a bad request, an auth failure, an exhausted retry with no generation id ever issued — and makes everything else degrade to the output already on hand instead of spending again.
A database constraint that was actively hostile to the feature it sat next to
PR #23 found a uniqueness constraint that made the sweep's entire purpose impossible at the schema level. A vendor-generation id was marked unique across the whole table, on the reasoning that one generation belongs to exactly one segment. But a rerun is specifically designed to reuse a prior run's generation ids — that's the entire point of recording them — so writing a reused id against a table-wide unique index threw a constraint violation that got silently swallowed as a warning. Measured against the three reruns that existed in production at the time: zero of them had successfully recorded a single shared id. Every one of them had re-attached to a prior generation for free in memory, written nothing durable about it, and was one interruption away from re-buying every segment it had just silently reused, at full price. The fix narrows the constraint to per-run uniqueness — the actual invariant worth enforcing — rather than per-table.
A receipt written too late to survive the exact failure it needed to survive
PR #24 found that a fallback generation's payment receipt was held in an in-memory Python list and only persisted to durable storage after the entire fallback block completed — after the poll, the download, and the length check. Every way a segment could die in between (a container redeploy mid-poll, an editor cancellation, an ffmpeg fault) lost the receipt for a generation that had already been bought, at up to roughly eighty cents a run. The primary generation path had followed the opposite, correct rule since it was first written — record the id the instant the vendor returns it, before the first poll — and the fallback path's own documentation claimed it did the same thing. It didn't. The fix records the id at the moment of purchase, and deliberately does not yet let the sweep act on fallback ids until a separate vendor-routing fix (below) makes probing them safe.
Guessing which vendor to ask is itself a bug
PR #34 is maybe the cleanest illustration in the whole arc of a sweep auditing a three-vendor pipeline while only being able to speak to one vendor. The query selected a vendor column and then discarded it, defaulting every probe to a single hardcoded API. Tested against one real, completed generation routed three different ways: asked correctly, it returned complete with a usable URL and the verdict "collect." Asked as either of the two wrong vendors, it came back not_found — read by the decision logic as "this generation is gone" — and the verdict was "give up." Two routings out of three threw away a paid, finished asset. The fix removes the default entirely: an unrecognized vendor or a missing credential now returns "could not ask," which is a different answer from "the generation doesn't exist," and the sweep is required to tell the two apart.
Honest messaging to the editor, not just a correct ledger
A smaller cluster (PR #29, #33, #35, #36) fixed what the editor actually sees once the sweep does its job correctly. A give-up verdict was being written to a field the run-detail panel never read, so an editor was told to keep waiting for — and explicitly not resubmit — a generation the system had just internally concluded was never coming. A successfully collected and finished run could still display a stale VendorQueuedTooLong error sitting right next to the word "Done," because nothing cleared the field once the generation was recovered. And a rerun's own cost-savings estimate was wrong in the specific population the whole sweep exists to serve: a segment only counted as "reusable" if every part of it was free, so a recovered generation with only its upscale step still pending read as zero savings, telling an editor "this is a full re-render" on a job that was actually 78% already paid for.
Operability and the parts that touch real money directly
The remaining PRs round out the picture: a confirmation dialog for stopping a parked run used to fire only once a segment had completed, which meant a run parked on its very first segment — with a generation already bought and in flight at the vendor — could be cancelled with one unconfirmed click, forfeiting money that cancellation (unlike collection) never recovers. A cost ledger was charging $0.00 for a completed, billed generation that had simply been rejected on quality grounds after the fact — real line items from production showed every other segment billing at the same consistent per-second rate while the rejected one logged as free. A quality audit that only checked whether an upscale URL was present missed an entire letterboxed, defective master shipping to an editor, because the two fallback outcomes it could distinguish both looked identical through that one narrow signal. And a logging bug meant five of seven real warning conditions in one production run never reached the log at all — a plain Python print statement reaches a platform's log aggregator classified only by which stream it came from, so warnings printed to stdout were filed as routine "info" lines sitting beside ordinary request chatter.
Woven through nearly every PR in this batch is a documentation discipline worth naming directly: a README was rewritten to describe files that no longer existed and omit the directory holding essentially all the maintained code; a gunicorn dependency got pinned to the exact version its configuration had been silently relying on, because a looser version range had already let one redeploy quietly resolve a new major and undo a prior log-level fix without anything alerting on it; and a one-write-per-poll bug inflated a 37-minute render into a database row that looked, from its timestamps alone, like it had taken 68 seconds — hundreds of identical writes whose only lasting effect was whichever one happened to land last.
The pattern: the auditor needs an auditor
Every bug in this arc shares the same shape. A background process was built specifically to catch money the main pipeline was losing — and it lost money the exact same way, for the exact same reasons: state that doesn't survive a restart, a constraint that doesn't model the real invariant, a default that stands in for a verdict nobody actually computed, a receipt written at the wrong moment to survive the failure it needed to survive. Building a reconciliation layer on top of a system doesn't automatically inherit correctness from good intentions. It's new code, with its own state machine and its own edge cases, and — as PR #21 proved outright — it can be built with such confidence that nobody notices it's structurally incapable of seeing the one case it exists to handle, until someone runs a high-effort review against real production data and checks.