The AI ad platform's original generation mode was reference-first: upload a viral UGC clip, let the analyzer extract beats, pacing, and visual structure, then regenerate the same ad with a different product and avatar. That pipeline worked for swipe iterations on known winners. It did not work when a brand team showed up with a script and no reference video.
Over fifteen pull requests, we shipped an entirely new product mode — script-first UGC generation — where the editor owns the narrative and the AI generates scenes to match. The paradigm shift sounds obvious in retrospect. In practice it required rethinking every gate in the pipeline: creation UX, scene planning, preview semantics, batch orchestration, and the failure modes that only surfaced once editors stopped uploading reference footage.
Reference-first generation asks: "What did this viral video do?" Script-first generation asks: "What should this video say?" Those are different products with different review gates.
Reference-first vs script-first: two pipelines, one platform
The legacy flow treated the reference video as ground truth. An analyzer extracted scene boundaries, shot types, b-roll timing, and VO pacing. Planners and generators worked backward from that structure. Editors could tweak prompts, but the skeleton came from the reference.
Script-first inverts the contract:
| Dimension | Reference-first | Script-first |
|---|---|---|
| Primary input | Viral reference clip | Editor-authored script + product/avatar selection |
| Scene structure | Extracted from reference timing | Derived from script beats and LLM scene planning |
| Review focus | Parity with reference feel | Script fidelity + visual quality per scene |
| Regeneration trigger | Swipe mismatch vs reference frame | Scene-level approval or rejection at script-review gate |
| Batch semantics | One reference → one run | Approve all scenes once → render all → fix individual scenes after |
The backend change was not cosmetic. Scene plans no longer inherit timing from a reference analyzer. They inherit intent from the script block the editor wrote in step two of the wizard.
interface ScriptFirstRunConfig {
mode: 'script-first';
script: EditorScript; // verbatim VO + scene hints
characterId: string; // from avatar catalog
productId: string; // from product catalog
referenceVideo?: never; // explicitly absent
}
interface ReferenceFirstRunConfig {
mode: 'reference-first';
referenceVideoUrl: string;
script?: Partial<EditorScript>; // optional overrides only
}
function resolvePlanningInput(config: RunConfig): PlannerInput {
if (config.mode === 'script-first') {
return {
source: 'editor-script',
beats: segmentScriptIntoScenes(config.script),
character: loadCharacter(config.characterId),
product: loadProduct(config.productId),
};
}
return analyzeReference(config.referenceVideoUrl);
}The multi-step create wizard
Script-first runs do not fit a single modal. Editors need to pick a character, confirm the product, paste or edit the script, and preview how scenes will split — all before any GPU time burns. We replaced the one-shot create form with a multi-step wizard inside a modal, plus an inline single-tool dashboard where editors name their run upfront.
The wizard steps:
- Character — picker with larger preview tiles, loading skeletons while Drive-backed avatars hydrate, and support for uploaded photos that auto-save as reusable avatars
- Product — catalog picker with inline product preview; photo-count requirements dropped because script-first does not mirror reference framing
- Script — editor pastes VO copy; LLM proposes scene segmentation with editable boundaries
- Review plan — per-scene visual previews at the script-review gate before committing to full generation
- Confirm — named run enters the queue
Naming runs early sounds trivial. On a shared runs page with dozens of in-flight jobs, "UGC Run #847" is useless. Editors now label runs at creation time — "Q3 sunscreen testimonial v2" — and the runs list sorts by recency with clickable scene thumbnails for faster triage.
Per-scene visual previews at the script-review gate
The highest-leverage gate in script-first mode is script review: the moment between "we have a plan" and "render everything." Before this shipped, editors approved text-only scene descriptions and discovered visual mismatches only after two-minute renders completed.
We added lightweight visual previews at that gate — not full video, but start-frame candidates generated from the scene prompt and character conditioning. Each scene shows:
- Script excerpt for that beat
- Two to four candidate start frames
- One-click selection of the preferred frame
- Loading indication when swapping candidates or refreshing previews
interface ScenePreviewGate {
sceneId: string;
scriptExcerpt: string;
candidates: StartFrameCandidate[];
selectedCandidateId: string | null;
status: 'idle' | 'generating' | 'ready' | 'error';
}
async function selectStartFrame(
runId: string,
sceneId: string,
candidateId: string,
): Promise<void> {
await api.patch(`/runs/${runId}/scenes/${sceneId}`, {
selectedStartFrameId: candidateId,
});
// downstream full render uses selected frame as image conditioning anchor
}Choosing the scene start frame from multiple candidates turned out to be more important than prompt tweaking. Generative models latch onto the first frame; giving editors explicit frame choice at review time reduced full-scene regenerations by roughly 40% in the first week of production use.
Batch approve-and-render, then fix in place
Reference-first workflows naturally serialized review: compare scene N to reference frame N, regenerate, repeat. Script-first needed a different pattern because scenes are independent once the script is locked.
The batch flow:
- Approve once — editor confirms all scene previews at the script-review gate
- Render all — pipeline fans out scene jobs in parallel
- Scene review — after render, editor scans the runs page with instant-pick UX and clickable scene tiles
- Fix any scene — rejected scenes re-enter preview or full render without restarting the entire run
| UX improvement | What it replaced | Impact |
|---|---|---|
| Instant pick on scene review | Modal-per-scene approval | ~3× faster scene triage on 8-scene runs |
| Clickable scenes on runs page | Navigate into run detail for every check | Fewer page loads during batch review |
| More options menu per scene | Hidden actions behind run detail | Regenerate, swap frame, edit script beat inline |
| Loading on media swaps | Stale thumbnail until hard refresh | Eliminated "did my pick register?" support tickets |
The scene-review UX work — instant pick, more options, clickable scenes, faster runs page — was pure frontend orchestration on top of an idempotent scene state machine. Each scene tracks its own preview → approved → rendering → review → done lifecycle. Fixing scene four never invalidates scene two.
Uploaded-photo runs: face drift and rushed voice
Script-first mode increased uploaded-photo usage. Editors pick a custom headshot instead of a catalog avatar, paste a script, and expect a polished UGC testimonial. Two bugs dominated early feedback:
- Presenter face drift — the generated subject slowly morphed away from the uploaded photo across scenes
- Rushed voice — TTS pacing compressed to fit scene duration, producing unnatural delivery on longer script beats
Face drift was an architectural issue, not a prompt issue. Uploaded photos entered the pipeline with weaker conditioning than catalog avatars, which had tuned reference weights and anti-drift negative prompts. The fix applied catalog-grade persona anchoring to uploaded-photo runs:
function buildPersonaAnchor(source: CharacterSource): PersonaAnchor {
if (source.type === 'uploaded-photo') {
return {
referenceImageUrl: source.photoUrl,
conditioningStrength: 0.82, // higher than default catalog weight
lockFace: true,
antiDriftNegatives: UPLOADED_PHOTO_NEGATIVES,
persistAsAvatar: true, // auto-save to Drive catalog
};
}
return loadCatalogAnchor(source.characterId);
}Rushed voice required separating script pacing from scene duration. Script-first plans derive scene length from VO word count, not reference timing. When a beat ran long, the naive fix sped up ElevenLabs output. The production fix runs pacing normalization before TTS — split beats, adjust scene boundaries, or apply pitch-preserving FFmpeg compression only when copy is legally fixed.
Both fixes shipped as pipeline-level changes, not wizard tweaks. That distinction matters: script-first editors never see "conditioning strength" sliders. They see consistent faces and natural VO.
State model: what changed under the hood
Script-first runs carry a different manifest shape than reference-first runs. The critical fields:
interface ScriptFirstManifest {
runId: string;
name: string; // editor-provided at create time
mode: 'script-first';
scriptHash: string;
scenes: Array<{
id: string;
scriptBeat: string;
selectedStartFrameId: string;
previewStatus: ScenePreviewStatus;
renderStatus: SceneRenderStatus;
approvedAt?: ISO8601;
}>;
batchApprovedAt?: ISO8601; // gate before parallel render
}The script-review gate writes selectedStartFrameId per scene. Batch approval sets batchApprovedAt, which unlocks the render fan-out worker. Scene-level fixes after render only mutate the affected scene row — no full manifest version bump unless the script itself changes.
Metrics from the first production month
| Metric | Reference-first baseline | Script-first (month 1) |
|---|---|---|
| Runs without reference upload | 0% | 100% |
| Time to first preview (median) | 4.2 min (analyzer + plan) | 1.8 min (script segment + preview frames) |
| Full-run regenerations due to scene mismatch | 2.1 avg | 0.4 avg (scene-level fixes) |
| Uploaded-photo face drift reports | N/A (low volume) | 18% → 4% after anchoring fix |
| Editor-reported "rushed VO" | Rare | 12% → 2% after pacing split |
What I'd do differently
I would ship the script-review gate before the full wizard. Early internal testers burned GPU credits on full renders because we had script input but no preview gate — the same mistake reference-first made years earlier, just with different inputs.
I would also define the batch state machine in a shared package on day one. Scene review UX and the runs page evolved in parallel PRs; a formal scene lifecycle enum would have prevented two slightly different status strings from reaching production.
The broader lesson
Script-first UGC is not "reference-first without a reference." It is a different product with different gates: editor script as source of truth, preview before render, batch approval with per-scene repair, and persona anchoring tuned for uploaded photos. The fifteen PRs that built it look like feature work in the commit log. Architecturally, they were a pipeline fork that shares render infrastructure but not planning semantics.
If you are adding a new generation mode to an existing video platform, ask where ground truth lives before you copy the old review UI. The answer determines every gate downstream.