Hardening a Headless API for Production CI/CD Integration

The platform's headless API lets external CI/CD pipelines trigger AI evaluations programmatically. When a third-party system depends on your API, reliability requirements go from "it usually works" to "it must always work."

FastAPI Lifespan Migration

FastAPI deprecated @app.on_event("startup") and @app.on_event("shutdown") in favor of the lifespan context manager. Beyond just following deprecation warnings, the lifespan pattern gives you proper resource cleanup guarantees — database connection pools, Redis connections, and background task schedulers all get clean shutdown instead of hoping __del__ fires.

# Before: deprecated event hooks
@app.on_event("startup")
async def startup():
    await init_db_pool()
    await init_redis()

@app.on_event("shutdown")
async def shutdown():
    await close_db_pool()

# After: lifespan context manager
@asynccontextmanager
async def lifespan(app: FastAPI):
    await init_db_pool()
    await init_redis()
    yield
    await close_redis()
    await close_db_pool()

app = FastAPI(lifespan=lifespan)

Langfuse Dataset-Trace Linking

The platform uses Langfuse for observability and evaluation tracking. The DatasetItem.link() method was throwing AttributeError because the SDK expected a trace object with specific attributes that our wrapper didn't expose. Fixed the integration to properly pass trace IDs, enabling dataset-level performance analysis across evaluation runs.

PR Metadata in Headless Flow

The headless API accepts evaluation requests from CI/CD. When a PR-based evaluation was triggered, the PR metadata (title, diff URL, file list) wasn't being passed to the evaluation agent. The agent was evaluating code without knowing which PR it came from — producing generic feedback instead of PR-specific analysis. Fixed the data flow to thread PR metadata through the entire evaluation pipeline.

Firestore Security Rules

The Firestore databases had security rules allowing unauthenticated read/write access. This was a critical vulnerability — anyone with the database URL could read or modify evaluation data. Enforced authentication on all Firestore rules, requiring valid IAM credentials for every operation.

// Before: open access
rules_version = '2';
service cloud.firestore {
  match /databases/{database}/documents {
    match /{document=**} {
      allow read, write: if true;
    }
  }
}

// After: authenticated access only
rules_version = '2';
service cloud.firestore {
  match /databases/{database}/documents {
    match /{document=**} {
      allow read, write: if request.auth != null;
    }
  }
}

Stuck-Task Detection

The headless stuck-task checker had a timezone TypeError — comparing timezone-naive Python datetimes with timezone-aware database timestamps. Same root cause as the task detail endpoint bug, but in a different code path. Also fixed an incorrect logging API call in the database layer that was swallowing error details.

Headless APIs that serve external systems need hardening that internal-only APIs can skip. Every fix here was about making the platform trustworthy for automated consumers that can't interpret ambiguous errors.