The AI video generation platform I work on does not bet on a single model vendor. Clip generation might route through Replicate one week and a direct Kling integration the next. Voiceover runs through ElevenLabs. Image conditioning hits fal. Planning and QA call Claude. Each vendor exposes a different async contract — webhooks, REST polling, SSE streams, queue IDs with opaque status enums.
When we had three vendors, each adapter shipped its own poll loop. Copy-paste with minor tweaks. When we hit five, the duplication became a reliability hazard: inconsistent timeout handling, no shared failure classification, and — most painfully — a watchdog process that killed jobs the poll loop claimed were hung but were actually still rendering on Replicate's GPU cluster.
I refactored all vendor polling into a single universal template with automatic heartbeats, configurable timeouts, exponential backoff, and structured failure classification. One abstraction, every model.
Five vendors, five polling dialects
Before the refactor, each vendor adapter owned its own waiting logic. The surface area looked like this:
| Vendor | Async model | Status API | Typical render time | Pre-refactor pain |
|---|---|---|---|---|
| Replicate | Prediction ID + poll | GET /predictions/:id | 90s–8min | No heartbeat → watchdog kill |
| Kling (direct) | Task ID + poll | POST status endpoint | 2–6min | Hard-coded 5min timeout too short |
| fal | Queue + webhook optional | GET queue status | 30s–3min | Webhook fallback never tested |
| ElevenLabs | Synchronous-ish TTS | Streaming response | 5–30s | Treated as sync, no poll wrapper |
| Claude | SSE stream | Stream events | 10–60s | Different error shape entirely |
Each row was a separate while loop with different sleep intervals, different terminal states, and different ideas of what "failed" meant. Replicate returns status: "failed" with an error string. Kling returns numeric error codes. fal returns HTTP 503 on queue saturation that should retry, not abort.
The watchdog kill incident
The production incident that forced this refactor: a background watchdog monitors long-running generation jobs. If a job doesn't emit a heartbeat within 120 seconds, the watchdog assumes the worker crashed and marks the job failed, releasing resources and notifying the portal.
Replicate clip renders routinely take three to eight minutes. The Replicate adapter's poll loop looked like this:
// BEFORE — no heartbeat, fixed 3s sleep, silent hang
async function pollReplicate(predictionId: string): Promise<ReplicateOutput> {
while (true) {
const prediction = await replicate.predictions.get(predictionId);
if (prediction.status === 'succeeded') return prediction.output;
if (prediction.status === 'failed') throw new Error(prediction.error);
if (prediction.status === 'canceled') throw new CancelledError();
await sleep(3000);
}
}The loop was alive — hitting Replicate's API every three seconds — but it never told the watchdog. After 120 seconds of silence, the watchdog killed a job whose clip was 60% rendered on Replicate's side. The operator saw "failed" in the portal. Replicate still charged for the completed render we never retrieved.
The fix was not "increase the watchdog timeout globally." That would mask actual worker crashes on fast vendors. The fix was: every poll loop must emit a heartbeat on every iteration, automatically, via shared infrastructure.
The universal poll template
The refactor extracted a generic pollUntilTerminal function that wraps any vendor-specific status check:
type PollStatus = 'pending' | 'running' | 'succeeded' | 'failed' | 'cancelled';
interface PollConfig<T> {
vendorName: string;
jobId: string;
checkStatus: () => Promise<{ status: PollStatus; result?: T; error?: VendorError }>;
timeoutMs: number;
initialIntervalMs: number;
maxIntervalMs: number;
backoffMultiplier: number;
onHeartbeat: () => void;
classifyError: (error: VendorError) => FailureClass;
}
type FailureClass = 'retryable' | 'fatal' | 'vendor_outage' | 'quota_exceeded';
async function pollUntilTerminal<T>(config: PollConfig<T>): Promise<T> {
const deadline = Date.now() + config.timeoutMs;
let interval = config.initialIntervalMs;
while (Date.now() < deadline) {
config.onHeartbeat(); // EVERY iteration, before the vendor call
const { status, result, error } = await config.checkStatus();
if (status === 'succeeded' && result !== undefined) return result;
if (status === 'cancelled') throw new CancelledError(config.jobId);
if (status === 'failed' && error) {
throw classifyAsPollError(config.classifyError(error), error);
}
await sleep(interval);
interval = Math.min(interval * config.backoffMultiplier, config.maxIntervalMs);
}
throw new PollTimeoutError(config.vendorName, config.jobId, config.timeoutMs);
}Vendor adapters shrink to two functions: one that maps vendor-specific API responses to the normalized PollStatus, and one that classifies vendor errors into retryable vs fatal. All timing, backoff, heartbeat, and timeout logic lives in the template.
After: Replicate adapter
// AFTER — 12 lines, heartbeat automatic, timeout configurable per model
async function pollReplicate(
predictionId: string,
ctx: JobContext,
): Promise<ReplicateOutput> {
return pollUntilTerminal({
vendorName: 'replicate',
jobId: predictionId,
checkStatus: async () => {
const p = await replicate.predictions.get(predictionId);
return {
status: mapReplicateStatus(p.status),
result: p.status === 'succeeded' ? p.output : undefined,
error: p.status === 'failed' ? { message: p.error, code: 'REPLICATE_FAIL' } : undefined,
};
},
timeoutMs: ctx.modelConfig.maxWaitMs ?? 600_000, // 10min for video models
initialIntervalMs: 2_000,
maxIntervalMs: 10_000,
backoffMultiplier: 1.5,
onHeartbeat: () => ctx.heartbeat.emit('poll:replicate'),
classifyError: classifyReplicateError,
});
}Every vendor adapter follows the same shape. Adding a sixth vendor means writing checkStatus and classifyError — not reinventing backoff math and heartbeat wiring.
Failure classification: retry vs abort vs escalate
Normalized polling is only half the value. The template also centralizes failure classification so upstream retry logic makes consistent decisions:
| Failure class | Examples | Pipeline behavior |
|---|---|---|
retryable |
HTTP 503, queue full, transient GPU unavailable | Retry with backoff, same vendor |
vendor_outage |
5xx sustained, status endpoint down | Fail over to alternate vendor adapter |
quota_exceeded |
Rate limit, billing cap, concurrency limit | Queue for later, notify operator |
fatal |
Invalid input, content policy violation, corrupt output | Abort scene, surface to operator with context |
Before classification was centralized, the Kling adapter retried content-policy failures three times (burning credits), while the Replicate adapter aborted immediately on the same class of error. Operators saw inconsistent behavior depending on which vendor routed the scene.
Exponential backoff without over-polling
Fixed-interval polling is either too aggressive (API rate limits on fast jobs) or too slow (added latency on jobs that finish between polls). The template uses exponential backoff with a cap:
- Initial interval: 2 seconds — catch fast TTS and image jobs quickly
- Backoff multiplier: 1.5× per iteration
- Max interval: 10 seconds — never go silent long enough to worry operators
- Heartbeat: every iteration regardless of interval — watchdog stays satisfied
For a six-minute video render, the poll cadence ramps from 2s → 3s → 4.5s → 6.75s → 10s (held). Total API calls: roughly 40, down from 120 at fixed 3-second intervals, with zero watchdog false positives.
Capping feel-regeneration runs
A related reliability fix shipped in the same arc: the "feel regeneration" path — which re-renders challenger clips to match reference pacing and energy — was regenerating every challenger on every quality pass with no cap.
A twelve-scene ad with three challengers per scene could trigger 36 extra video renders on a single feel-regen pass. Most of those rerenders targeted challengers that were already within quality threshold — the regen was running primary-only checks against all challengers indiscriminately.
interface FeelRegenPolicy {
scope: 'primary-only'; // don't regen challengers unless primary fails
maxRegensPerRun: number; // hard cap, default 3
qualityThreshold: number; // skip regen if score >= threshold
}
async function runFeelRegen(
scenes: SceneResult[],
policy: FeelRegenPolicy,
): Promise<SceneResult[]> {
let regenCount = 0;
const updated: SceneResult[] = [];
for (const scene of scenes) {
const primary = scene.challengers.find(c => c.role === 'primary');
if (!primary || primary.feelScore >= policy.qualityThreshold) {
updated.push(scene);
continue;
}
if (regenCount >= policy.maxRegensPerRun) {
updated.push(scene); // cap hit — ship best available
continue;
}
const regen = await regenPrimary(scene, primary);
regenCount++;
updated.push(applyRegen(scene, regen));
}
return updated;
}Primary-only scope plus a per-run cap of three regenerations cut feel-regen GPU spend by roughly 70% without measurable quality regression — because most challenger rerenders were fixing problems the primary didn't have.
Before and after: what changed in production
| Metric | Before universal template | After |
|---|---|---|
| Watchdog false kills (Replicate jobs) | ~4 per day | 0 |
| Poll loop code (lines across vendors) | ~340 duplicated | ~80 template + ~25 per vendor adapter |
| Mean time to diagnose vendor timeout | 45 min (which adapter?) | 5 min (structured logs with vendor + jobId) |
| Feel-regen GPU calls per run | Up to 36 | Max 3 (primary-only) |
Design principles for multi-vendor async
If you're integrating multiple AI model providers into one pipeline, these rules saved us repeated incidents:
- Normalize early. Map vendor status enums to your own terminal states at the adapter boundary, not in business logic.
- Heartbeat is not optional. Any loop that can run longer than your watchdog interval must emit heartbeats — bake it into the template so adapter authors can't forget.
- Timeout per model, not per vendor. Video generation gets ten minutes. TTS gets two. Image gen gets five. Store timeouts in model config, not hard-coded in adapters.
- Classify failures once. Upstream retry and failover logic should consume
FailureClass, not parse vendor error strings. - Cap expensive recovery paths. Regeneration, failover, and retry loops all need per-run budgets or costs compound silently.
The universal poll template isn't glamorous infrastructure. Nobody demos it to investors. But it's the difference between a pipeline that survives vendor outages gracefully and one that burns credits on phantom failures while the watchdog murders live jobs.