Splicing AI-generated sections into existing video is easy. Making the splice invisible is not. When Variant Multiplier swaps a body section or Hook Machine stitches a generated opener onto A-roll, the viewer's eye catches the cut — a colour temperature shift, a brightness jump, a frame where motion stutters. This post covers the three techniques that repair those seams: deterministic grade-match, end-frame interpolation, and word-level karaoke captions.
Neural colour grading is impressive in demos. Deterministic histogram matching is what you ship when you need the same splice to look identical on the thousandth run.
The visual seam problem
AI video engines render with their own implicit colour pipeline — white balance, contrast curve, saturation profile. Source footage from a real UGC shoot carries the phone camera's pipeline. Concatenating them with FFmpeg's concat demuxer produces a visible discontinuity at every boundary:
- Colour shift — generated section is cooler/warmer than source
- Exposure jump — generated section is brighter/darker
- Motion discontinuity — subject position does not match at cut-back
- Caption desync — text timing does not align with new VO
| Seam type | Viewer perception | Repair technique |
|---|---|---|
| Colour discontinuity | "This looks edited / fake" | Grade-match |
| Motion jump at cut-back | "Something glitched" | End-frame interpolation |
| Caption timing drift | "Subtitles are wrong" | Word-level karaoke + libass |
| Audio pop at boundary | "Bad edit" | Crossfade + silence alignment |
Grade-match: deterministic colour transfer
feat/grade-match implements histogram-based colour transfer from a reference frame (the last frame of the preceding source section) to the first and last frames of the generated section. The algorithm:
- Extract reference frame RGB histogram from source video at splice point
- Extract target frame histogram from generated section boundary
- Compute per-channel lookup table mapping target → reference distribution
- Apply LUT to all frames in the generated section
interface ChannelLUT {
r: Uint8Array; // 256-entry lookup
g: Uint8Array;
b: Uint8Array;
}
function computeGradeMatchLUT(
referenceFrame: RGBBuffer,
targetFrame: RGBBuffer,
): ChannelLUT {
const refHist = { r: histogram(referenceFrame.r), g: histogram(referenceFrame.g), b: histogram(referenceFrame.b) };
const tgtHist = { r: histogram(targetFrame.r), g: histogram(targetFrame.g), b: histogram(targetFrame.b) };
return {
r: matchHistograms(tgtHist.r, refHist.r),
g: matchHistograms(tgtHist.g, refHist.g),
b: matchHistograms(tgtHist.b, refHist.b),
};
}
function applyLUT(frame: RGBBuffer, lut: ChannelLUT): RGBBuffer {
return {
r: frame.r.map(v => lut.r[v]),
g: frame.g.map(v => lut.g[v]),
b: frame.b.map(v => lut.b[v]),
};
}The FFmpeg integration wraps this as a filter graph applied before concat:
# Grade-match filter applied to generated section before concat
ffmpeg -i generated_section.mp4 -i reference_frame.png \
-filter_complex "[0:v]lut3d=grade_match.cube[graded];[graded][1:v]..." \
-c:v libx264 -crf 18 graded_section.mp4Why deterministic beats neural for colour transfer
| Property | Histogram grade-match | Neural colour grading |
|---|---|---|
| Reproducibility | Identical output every run | Stochastic — varies per inference |
| Latency | <200ms per section | 2–8s per frame (GPU inference) |
| Cost | CPU-only, $0 | GPU/API cost per frame |
| Hallucination risk | None — pixel-level LUT | Can alter texture, add artefacts |
| Debuggability | LUT is inspectable artifact | Black box |
Neural colour grading excels when you need creative grading — "make this look like golden hour." Splice repair needs matching, not creativity. Histogram transfer solves matching without introducing artefacts.
End-frame interpolation
Grade-match fixes colour. It does not fix motion. When a generated section ends and the stitch cuts back to original source footage, the subject's head position, hand gesture, and background motion rarely align. feat/end-frame generates a transition frame that bridges the gap.
The pipeline:
- Extract last frame of generated section (after grade-match)
- Extract first frame of following source section
- Generate interpolated frame(s) blending pose/lighting between the two
- Insert 2–4 interpolated frames (80–160ms at 25fps) before the hard cut-back
interface EndFrameConfig {
interpolationFrames: number; // typically 2–4
method: 'optical-flow' | 'cross-dissolve' | 'model-interp';
}
async function buildEndFrameTransition(
generatedLastFrame: Buffer,
sourceFirstFrame: Buffer,
config: EndFrameConfig,
): Promise<Buffer[]> {
if (config.method === 'cross-dissolve') {
return linearCrossDissolve(generatedLastFrame, sourceFirstFrame, config.interpolationFrames);
}
// optical-flow or model-based for higher quality when motion delta is large
return opticalFlowInterp(generatedLastFrame, sourceFirstFrame, config.interpolationFrames);
}Cross-dissolve handles small motion deltas (talking head, minimal gesture change). Optical-flow interpolation handles larger deltas but costs more compute. The stitch pipeline selects method based on motion magnitude between boundary frames.
Cut-back smoothness metrics
| Configuration | Visible seam rate (internal QA) | Added latency |
|---|---|---|
| Raw concat (no repair) | 78% | 0ms |
| Grade-match only | 34% | ~150ms |
| Grade-match + 2-frame dissolve | 12% | ~200ms |
| Grade-match + optical-flow interp | 4% | ~800ms |
Word-level transcript selection
Seam repair is visual, but splice accuracy starts with temporal precision. feat/word-level-selection provides karaoke-style word boundaries that define exactly where the generated section begins and ends in the source timeline.
interface WordSpan {
word: string;
startMs: number;
endMs: number;
confidence: number;
}
function buildSpliceTimeline(
words: WordSpan[],
sectionStart: number,
sectionEnd: number,
generatedDurationMs: number,
): StitchTimeline {
const preRoll = words.slice(0, sectionStart);
const postRoll = words.slice(sectionEnd + 1);
return {
segments: [
{ type: 'source', wordSpan: preRoll },
{ type: 'generated', durationMs: generatedDurationMs },
{ type: 'source', wordSpan: postRoll },
],
splicePoints: [
{ ms: preRoll.at(-1)?.endMs ?? 0, repair: ['grade-match', 'end-frame'] },
{ ms: preRoll.at(-1)?.endMs ?? 0 + generatedDurationMs, repair: ['grade-match', 'end-frame'] },
],
};
}Word-level boundaries prevent the most common splice error: cutting mid-phoneme. Silence-based selection gets close; word selection gets exact.
Karaoke captions: libass with drawtext fallback
After splicing, captions must reflect the new timeline — including regenerated VO in the swapped section. feat/karaoke-captions renders word-highlighted captions synchronized to the final stitched audio.
libass rendering (primary path)
libass renders ASS subtitles with per-word highlight timing — the "karaoke" effect where each word lights up as spoken. This requires font files in the container and FFmpeg compiled with libass support.
function generateAssKaraoke(words: WordSpan[], style: CaptionStyle): string {
const lines = ['[Script Info]', 'Title: Karaoke', '', '[V4+ Styles]', `Style: Default,${style.font},${style.fontSize},...`, '', '[Events]', 'Format: Layer, Start, End, Style, Text'];
for (const word of words) {
const start = msToAssTime(word.startMs);
const end = msToAssTime(word.endMs);
lines.push(`Dialogue: 0,${start},${end},Default,{\\c&H${style.highlight}&}${word.word}`);
}
return lines.join('\n');
}drawtext fallback
When libass is unavailable (minimal Docker images, missing fonts), the pipeline falls back to FFmpeg's drawtext filter — one word at a time, positioned bottom-center:
ffmpeg -i stitched.mp4 -vf "
drawtext=text='{{word}}':fontfile=/fonts/Inter-Bold.ttf:
fontsize=48:fontcolor=white:borderw=2:bordercolor=black:
x=(w-text_w)/2:y=h-120:enable='between(t,{{start}},{{end}})'
" -c:v libx264 -c:a copy captioned.mp4drawtext lacks the smooth highlight animation of libass but guarantees captions render in any environment. The fallback is automatic — runtime detection, no config flag.
| Renderer | Visual quality | Dependencies | Render time (30s video) |
|---|---|---|---|
| libass | Smooth karaoke highlight | libass + font files | ~3s |
| drawtext | Word-at-a-time, no highlight | FFmpeg only | ~8s (filter chain overhead) |
Composing the splice repair stack
In production, all three techniques compose in a fixed order:
- Word-level selection — define precise splice boundaries
- Grade-match — colour transfer on generated section
- End-frame interpolation — motion bridge at cut-back points
- Concat — FFmpeg stitch with audio crossfade
- Karaoke captions — render on final stitched output
async function spliceWithRepair(
source: VideoAsset,
generated: VideoAsset,
section: SectionBounds,
words: WordSpan[],
): Promise<StitchedOutput> {
const refFrame = await extractFrame(source, section.startMs - 1);
const graded = await gradeMatch(generated, refFrame);
const endFrames = await buildEndFrameTransition(
await extractFrame(graded, graded.durationMs - 1),
await extractFrame(source, section.endMs),
);
const stitched = await concatSegments([source.before(section), graded, ...endFrames, source.after(section)]);
const captioned = await renderKaraokeCaptions(stitched, realignWords(words, section));
return captioned;
}What I'd do differently
I would compute grade-match LUTs once per source video and cache them keyed by source frame hash. Currently each section swap recomputes histograms from scratch — cheap individually, but redundant in batch queue runs on the same source.
I would also expose seam quality as a numeric score in the UI, not just pass/fail. Editors could choose to accept a 12% seam risk to save 800ms of optical-flow interpolation when shipping variants quickly.
The pattern: repair at the splice, not in the generator
Video generation models will get better at colour consistency over time. They will not solve splice repair — the source footage's colour pipeline is external to any model. Building deterministic repair as a post-generation stitch step decouples splice quality from model quality and keeps the stack reproducible at production scale.