Blog

Vision-in-the-Loop: When the AI Rewrites Its Own Prompts from the Generated Frame

The AI generates a frame, inspects its own output with vision, and rewrites the prompt to fix camera, color, or composition issues.

Universal Vendor Polling: One Heartbeat Template for Every AI Model Provider

A single polling abstraction with heartbeats, timeouts, and exponential backoff that wraps every AI vendor's status endpoint.

Post-Launch Audit: 20 PRs and 12 Fixes in 48 Hours

Systematic post-launch triage across reliability regressions, UX confusion, and data honesty — batched into themed fix rounds.

Streaming AI-Generated Video from Cloud Storage to Browser

Range-request streaming from Google Drive through Next.js route handlers, index-first run history, and the Safari black-video fix.

Validate Before Building: A Technical Discovery Framework for Startup Clients

How a discovery call pivoted from an 8-phase fixed-price MVP build to problem validation first — and why refusing to design immediately built more trust.

Building an AI Video Generation Pipeline from Scratch

Five-stage pipeline: analyzer, Claude planner, Kling clip gen, FFmpeg stitch with sidechain-ducked audio, and cross-scene narrative QA.

Viral Feel Parity: Making AI-Generated Ads Feel Like the Original

Avatar consistency, sidechain-ducked audio beds, emotion-driven VO, caption rendering, color grading, and L/J-cut transitions.

Narrative Continuity: Story Spine and L/J-Cut Editing in AI Video

Story spine enforcement, cross-scene narrative QA, continuous VO over cuts, and pacing auto-fit.

Production Hardening an AI Video Pipeline: Retries, Fallbacks, and Crash Guards

Transient error classifier with exponential backoff, Claude model fallback, process crash guards, and Zod null tolerance.

QC Orchestration: Agent-Driven Quality Checks in a Review Pipeline

Config-driven QC service dispatching to external AI agents with HMAC-signed webhooks and state machine integration.

Halving Reviewer Clicks: Feedback Navigator, Keyboard Nav, and Read-Only Context

51% click reduction with chip-based feedback navigator, keyboard shortcuts, and a reusable read-only context pattern.

From Fixed Levels to L(n): Building an Extensible Taxonomy System

Replaced a rigid 2-level hierarchy with a config-driven L(n) taxonomy — YAML seeding, GCS CAS writes, and admin CRUD UI.

When RAG Says Duplicate but the LLM Disagrees: Building an Adjudication Layer

Two-threshold band architecture where an LLM second-opinion overrides RAG false positives.

Building an Events Table for Pipeline Observability

Unified events timeline aggregating four sources into one diagnostic view with causal ordering.

Redesigning Bulk Intake: Cascading Dropdowns and Taxonomy Validation

Replaced a checkbox taxonomy grid with cascading dropdowns and server-side slug validation.

Dead Letter Queues for Pipeline Reliability

DLQ escalation with reason codes, inline and sweep detection, GCS manifest fields, and UI badges.

When Two Cache Layers Serve Stale Data

Server-side unstable_cache plus client-side gcTime — a two-part cache debugging story.

A UX Polish Sprint: 8 PRs in 48 Hours

Filter registry, consolidated controls, search validation — eight small PRs compounding into a polished dashboard.

GCS Race Conditions: Generation-Fenced Lease Deletion

Two race conditions fixed with GCS generation-based optimistic concurrency.

Multi-Layer Caching for Dashboard Latency

HTTP Cache-Control, server-side cache with tag invalidation, React Query migration.

Pipeline Hardening: Timeouts, Traceability, and Graceful Shutdown

Dispatch registry for configurable step timeouts, SIGTERM graceful shutdown, dynamic lease TTL.

Building a GCS-Backed Auto-Seeder Platform

Provider-service architecture with hot-swappable backends and run manifests.

When 57K Lines Get Rolled Back

11 stacked PRs merged as one commit, rolled back, then reconsolidated with strict layering.

Invisible Bugs: File Type Bypass and Missing Worker Runs

Two production bugs invisible during development — file type validation bypass and missing worker runs.

Designing a Task Lifecycle V1 from Scratch

10-table schema with composite keys, immutable versioning, and 40+ API routes.

Building an SFT Recording Pipeline

Docker sandboxes with VNC, trajectory capture, video recording, and review workflows.

When Display Labels Break Sorting

Three bugs in rank computation, date tiebreakers, and deterministic windowing.

Taming a State Machine: Bulk Admin Ops

Transition matrices, workflow decoupling, unified status resolution, and JSON upload.

Decoupling Claim Timeouts from Feature Flags

Three-phase countdown UX with urgent and expired states.

Bootstrapping Engineering Standards on a Legacy Codebase

AI tool rules, PR templates, testing mandates, and CI upgrades — layered incrementally.

Production-Ready Multi-Turn Evaluation

Harness testing, snapshots via base64 transfer, crash-resilient resume state.

Reverse-Engineering a Codebase Into Architecture Docs

Mermaid data flow diagrams, conditional logic tables, and three verification passes.

Audit-Safe Admin Tools with Event Sourcing

Atomic Prisma transactions, event sourcing for audit trails, and preview mode.

Multi-Turn Agent Evaluation: Persistent State

Persistent Docker environments, conversation memory, and progressive hint injection.

Docker Image Hardening for AI Benchmarks

Pre-flight validation, runtime layer injection, and skip lists for multi-language images.

Adding Responses API to an Agent Framework

Auto-detection, input conversion, tool schema flattening, and response normalization.

Dev Starter Kit: AI Coding Tools as Senior Engineers

CI/CD workflows, test coverage enforcement, and pre-configured AI tool instructions.

Building an AI Evaluation Platform on GCP

Multi-cluster system spanning Cloud Run, GKE, and managed databases.

Event-Driven Architecture on GCP: Pub/Sub

6-phase pipeline with Pub/Sub topics, dead-letter queues, and Cloud Scheduler.

Debugging AlloyDB SSL Connection Drops

Random SSL connection failures causing 5% task failure rate — fixed with 3 config lines.

Idempotent Cloud Tasks Handlers in Python

FastAPI, PostgreSQL, and deterministic UUIDs for at-least-once delivery systems.

Zero-Downtime Embedding Migration

Switching embedding models in production using pgvector with zero downtime.

RL Training Arena for Code Agents

Gym-style API, Cloud Build sandboxing, Redis episode state, and trajectory recording.

Security Audit: 25 Critical Issues

SQL injection, hardcoded credentials, and secrets in production logs.

Building the Core AI Evaluation Engine

AlloyDB SSL fixes, KEDA autoscaling, Firestore misrouting, and headless API.

Automated LLM Scoring Service

Idempotent Cloud Tasks, LiteLLM proxy, Redis state, and VPC networking.

Auth Gateway with Admin Dashboard

Google OIDC, JWT validation, Kong integration, and notifications admin.

Notification Service: Event Delivery

Cloud Tasks delivery, circuit breakers, dead-letter queues, and webhook dispatch.

RAG Retrieval Service: pgvector and Embeddings

Similarity search, emergency embedding migration, and provenance tracking.

RL Arena Executor and Preprocessor

Gym-style API, Cloud Build sandboxing, Redis episode management.

Workflow, Taxonomy, and Platform Services

HITL workflow orchestration, taxonomy-based routing, and JWT auth fixes.

Incident Response and CLI Agent Debugging

Runbooks with 70+ diagnostic commands and sandbox environment fixes.

Systematic Service Hardening

N+1 queries, race conditions, SSRF vulnerabilities, and 124 new tests.

Production Bug Hunting: 33 Issues

5-layer audit methodology — path traversal, datetime bugs, CORS misconfigurations.

CI/CD Quality Gates That Actually Catch Bugs

Test coverage enforcement, dynamic test discovery, and CI gates.

When Your Logging Framework Crashes Production

Crash-proof logging infrastructure with safe_log_extra helpers.

Hardening a Headless API for Production

FastAPI lifecycle hooks, Langfuse integration, and Firestore security rules.