Streaming AI-Generated Video from Cloud Storage to Browser

A finished AI-generated ad on the video generation platform I work on is not a lightweight asset. A twelve-scene stitch with captions, mixed audio, and color grade routinely hits 500MB–1.2GB. Per-scene clips before stitch are 20–80MB each. The portal needs to preview these in the browser, export them for download, and serve them during active generation — all while artifacts live in Google Drive, not on the application server's local disk.

The naive approach — download the entire file into memory, then send it to the client — failed in three distinct ways: out-of-memory crashes on serverless workers, multi-second delays before the first byte reached the browser, and Safari rendering a black rectangle instead of video for certain H.264 profiles. This post covers the streaming architecture that fixed all three.

The buffering trap

The original media serving path looked like this:

1. Client requests /api/media/final?runId=abc123
2. Server resolves the Google Drive file ID from run metadata
3. Server downloads the entire file into a Buffer
4. Server responds with Content-Length and the full body

For a 200MB clip on a Railway worker with 512MB RAM, step three consumed 40% of available memory per concurrent request. Two operators previewing different runs simultaneously triggered OOM kills. On Vercel serverless functions with a 50MB response limit, the approach simply couldn't work for final renders at all.

Time-to-first-byte was equally bad. The browser showed a blank video element for eight to fifteen seconds while the server fetched the complete file from Drive before sending a single byte downstream.

File size	Buffer download (Drive → server)	TTFB to browser	Memory per request
50MB scene clip	~2.1s	~2.3s	50MB
200MB partial stitch	~8.4s	~8.7s	200MB
800MB final render	~31s	~31s (or OOM)	800MB

Streaming route handlers with Range support

The fix replaced buffer-then-send with a streaming proxy that pipes Drive bytes through a Next.js route handler, honoring HTTP Range requests so the browser can seek without downloading the full file.

// app/api/media/stream/route.ts
import { google } from 'googleapis';
import { NextRequest } from 'next/server';

export async function GET(req: NextRequest) {
  const fileId = req.nextUrl.searchParams.get('fileId');
  if (!fileId) return new Response('Missing fileId', { status: 400 });

  const drive = getDriveClient();
  const meta = await drive.files.get({ fileId, fields: 'size,mimeType,name' });
  const totalSize = parseInt(meta.data.size ?? '0', 10);
  const mimeType = meta.data.mimeType ?? 'video/mp4';

  const rangeHeader = req.headers.get('range');

  if (rangeHeader) {
    const { start, end } = parseRange(rangeHeader, totalSize);
    const stream = await drive.files.get(
      { fileId, alt: 'media' },
      {
        headers: { Range: `bytes=${start}-${end}` },
        responseType: 'stream',
      },
    );

    return new Response(stream.data as ReadableStream, {
      status: 206,
      headers: {
        'Content-Type': mimeType,
        'Content-Range': `bytes ${start}-${end}/${totalSize}`,
        'Content-Length': String(end - start + 1),
        'Accept-Ranges': 'bytes',
        'Cache-Control': 'private, max-age=3600',
      },
    });
  }

  // Full file stream (no buffering)
  const stream = await drive.files.get(
    { fileId, alt: 'media' },
    { responseType: 'stream' },
  );

  return new Response(stream.data as ReadableStream, {
    status: 200,
    headers: {
      'Content-Type': mimeType,
      'Content-Length': String(totalSize),
      'Accept-Ranges': 'bytes',
      'Cache-Control': 'private, max-age=3600',
    },
  });
}

Key properties of this handler:

• Never buffers the full file — Drive's readable stream pipes directly to the Response body
• Range requests return 206 Partial Content — browser video elements send Range headers for seek/scrub
• Accept-Ranges: bytes advertised on full responses so clients know seeking is supported
• Same handler for disk and Drive — local dev reads from filesystem with identical Range semantics via fs.createReadStream

After deployment, TTFB for an 800MB final render dropped from 31 seconds to under 400 milliseconds — the time to establish the Drive stream and return headers, not download the file.

Retiring rewrite proxies

An earlier approach used Next.js rewrites to proxy /generate and /export paths to the backend worker. Rewrites buffer responses internally and don't expose Range request control to the route author. The streaming route handlers replaced those rewrites entirely:

Path	Old (rewrite)	New (streaming handler)
/api/generate/preview	Rewrite → worker, full buffer	Stream from Drive/disk with Range
/api/export/download	Rewrite → worker, full buffer	Stream from Drive/disk with Range
/api/media/stream	Did not exist	Universal stream endpoint (fileId param)

Centralizing on one stream handler reduced the surface area: every video element in the portal points at /api/media/stream?fileId=... regardless of whether it's a scene clip, partial stitch, or final export.

Index-first run history

Streaming fixed playback. A separate performance problem blocked the run-history page: listing past runs required scanning the entire Google Drive folder tree before rendering a single table row.

The old flow:

1. Operator opens run history
2. Server lists all Drive folders under /runs/
3. For each folder, server reads manifest.json to get metadata
4. Table renders after full scan completes (8–20 seconds)

The index-first pattern decouples listing from Drive scanning:

// Index lives in DB — populated at run creation, updated at completion
interface RunIndexEntry {
  id: string;
  name: string;
  status: RunStatus;
  createdAt: Date;
  sceneCount: number;
  driveFolderId: string;
  thumbnailFileId?: string; // populated async after first frame generates
}

async function listRunHistory(page: number): Promise<RunHistoryPage> {
  // Immediate: render table from DB index
  const entries = await db.runIndex.list({ page, pageSize: 25 });

  // Lazy: thumbnails resolved client-side via stream endpoint
  return {
    runs: entries.map(e => ({
      ...e,
      thumbnailUrl: e.thumbnailFileId
        ? `/api/media/stream?fileId=${e.thumbnailFileId}`
        : null,
    })),
    // Background reconciliation — never blocks the response
    reconcilePromise: reconcileDriveIndex().catch(logReconcileError),
  };
}

The table renders immediately from the DB index. Thumbnail images load asynchronously via the streaming endpoint as each thumbnailFileId resolves. A background reconciliation job periodically compares the Drive folder tree against the index and surfaces orphan runs — but never blocks the operator's page load.

Run-history time-to-interactive went from 8–20 seconds to under 300ms.

Safari black-video fix

Even with streaming working, Safari showed a black video element for certain generated clips. Chrome and Firefox played the same files without issue. Two root causes:

Wrong poster frame

Video elements used the reference (competitor) keyframe as the poster attribute while the src pointed at the generated clip. Safari renders the poster until sufficient video data arrives to decode the first frame. When the poster and video content had different dimensions or color profiles, Safari occasionally failed to transition from poster to video — leaving a black box.

Fix: poster frames now come from the generated content's first keyframe, extracted during stitch and stored alongside the clip in Drive.

<!-- BEFORE: competitor keyframe as poster -->
<video
  src="/api/media/stream?fileId=generated-clip-id"
  poster="/api/media/stream?fileId=reference-keyframe-id"
/>

<!-- AFTER: generated frame as poster -->
<video
  src="/api/media/stream?fileId=generated-clip-id"
  poster="/api/media/stream?fileId=generated-poster-id"
  preload="metadata"
/>

Codec profile mismatch

Some AI video model providers output H.264 High Profile Level 5.1. Safari on older macOS versions supports High Profile up to Level 4.2 in hardware decode. The streaming fix didn't help — the bytes arrived fine, but the decoder rejected them.

The stitch pipeline now re-encodes final output to H.264 High Profile Level 4.0 with yuv420p pixel format during FFmpeg assembly — compatible with Safari hardware decode while preserving visual quality at ad-delivery bitrates.

# Stitch output re-encode for Safari compatibility
ffmpeg -i concat_list.txt \
  -c:v libx264 -profile:v high -level 4.0 \
  -pix_fmt yuv420p \
  -c:a aac -b:a 192k \
  -movflags +faststart \
  final_output.mp4

+faststart moves the moov atom to the beginning of the file, which matters for streaming — Safari needs metadata before it can begin decode, and with moov at the end, even a Range-capable stream can't start playback until the index is fetched.

Architecture overview

Layer	Responsibility	Latency target
DB run index	Metadata for history table	<100ms query
Stream route handler	Range-aware proxy from Drive/disk	<400ms TTFB
Client video element	Progressive download + seek via Range	Playback starts at ~1MB buffered
Background reconcile	Drive folder scan vs index drift	Async, non-blocking
FFmpeg stitch	Safari-safe encode + faststart	At generation time, not serve time

Results

Metric	Before	After
TTFB (800MB final render)	~31s	~380ms
Memory per concurrent stream	Full file size	~64KB (stream buffer)
Run history time-to-interactive	8–20s	<300ms
Safari black-video reports	~15% of previews	0%
OOM kills on media requests	2–3 per week	0

When to stream vs when to CDN

Streaming through the app server is the right first step when files live in private Google Drive folders with per-user access control — you can't put a signed CDN URL in front of Drive without an additional sync layer. The stream handler acts as an authorization gate: it checks run ownership before piping bytes.

The next evolution — if preview traffic scales beyond what Railway workers handle — is syncing completed renders to GCS with signed URLs and putting Cloud CDN in front. The stream handler's Range logic transfers directly; only the upstream source changes from Drive API to GCS bucket. The index-first pattern and Safari-safe encode remain unchanged.

For AI video platforms where outputs are large, private, and frequently seeked during review, streaming isn't an optimization — it's a requirement. Buffering worked when clips were 5MB proofs of concept. At production scale, the pipe has to flow in both directions without filling the bucket.