Building a GCS-Backed Auto-Seeder Platform

The team needed a web-based seeding platform to replace a desktop tool for preparing AI evaluation tasks. The desktop tool had years of muscle memory behind it, and it stored everything in Google Cloud Storage as JSON manifests. A browser experience had to land quickly on that same data, but we also knew the long-term home for task metadata was a relational database with a proper service layer. Building the first version directly on Prisma would have blocked product work for months while schemas and migrations caught up to a moving task system. Building only against GCS would have produced a pile of throwaway controller code the moment we flipped storage.

This post is about the architecture that let us ship on GCS immediately without painting ourselves into a corner: routes that stay thin, services that encode business rules once, and providers that swap between object storage and SQL behind a single interface. Along the way we added a run manifest store, a versioned API adapter for the Next.js frontend, registries for artifacts and pipeline steps, a thirteen-node lifecycle graph with gates, and a task detail experience that matches the desktop tool closely enough that operators do not need retraining.

The problem in one sentence

We had to satisfy today’s reality (JSON in GCS) and tomorrow’s reality (normalized rows in Postgres) without writing the product twice or blocking the roadmap on database readiness.

Provider-service architecture

The fix is a three-layer shape: routes → services → providers. HTTP handlers in Next.js validate input, attach request context, and delegate. Services own business logic—creating runs, resolving step identifiers, assembling views for the UI—and they depend only on a TypeScript interface that describes persistence. The concrete GcsProvider implements that interface by reading and writing objects in a bucket; when the schema and Prisma models are ready, a PrismaProvider implements the same methods against the database.

Services and routes do not branch on storage. They call runManifest.create, taskVersion.get, or whatever the interface exposes. Swapping providers is configuration: an environment flag or dependency injection at the composition root. That is the difference between “we are waiting on migrations” and “we already shipped; migration is an implementation detail.”

export interface TaskStorageProvider {
  getTaskVersion(taskId: string, version: string): Promise<TaskVersionRecord | null>;
  writeRunManifest(taskId: string, manifest: RunManifest): Promise<void>;
  listRuns(taskId: string): Promise<RunSummary[]>;
}

export class GcsTaskStorageProvider implements TaskStorageProvider {
  constructor(private bucket: BucketName, private client: GcsClient) {}

  async getTaskVersion(taskId: string, version: string) {
    const path = `tasks/${taskId}/versions/${version}.json`;
    return readJsonFromGcs<TaskVersionRecord>(this.client, this.bucket, path);
  }

  async writeRunManifest(taskId: string, manifest: RunManifest) {
    const path = `tasks/${taskId}/.manifest.json`;
    await writeJsonToGcs(this.client, this.bucket, path, manifest);
  }

  async listRuns(taskId: string) {
    // prefix list + parse manifests
    return [];
  }
}

The important discipline is naming methods after capabilities the product needs, not after GCS verbs. If the interface stays honest, Prisma never leaks $transaction into routes and GCS never leaks bucket paths into services beyond the provider.

Run manifest store

Each task run is represented as a single JSON document in the bucket at tasks/{taskId}/.manifest.json. That convention matches what the desktop tool already produced: one manifest per task folder, easy to inspect with standard tools, and simple to list with a prefix query. RunManifestService exposes create, list, get, and update operations; the service validates invariants (for example, monotonic step progression or required metadata fields) and calls the provider.

We already had a readJsonFromGcs helper for pulling typed JSON from objects. Symmetry demanded a writeJsonToGcs companion: same serialization options, same error mapping, explicit content types for debugging. Providers stay small; shared I/O helpers keep retry and logging consistent whether the caller is writing a manifest or a sidecar artifact.

V1 adapter layer

The React client was written against stable domain types—call them AutoSeed* types—shaped for forms, tables, and step viewers. The HTTP API backed by GCS naturally returns versioned wire shapes—here, V1TaskVersion—because the bucket layout and field names evolved with the desktop tool. Coupling the UI to those wire types would have made every storage migration a frontend rewrite.

The compromise is a client-side adapter, adaptV1TaskVersion, that maps API responses into the UI’s domain model. When the backend eventually serves Prisma-backed JSON, only the server route and possibly a thin server adapter change. The browser keeps importing the same AutoSeedTaskVersion types and the same components. The adapter is the pressure valve between “what the storage layer can emit today” and “what the product wants to render.”

type AutoSeedTaskVersion = {
  id: string;
  pipelineStepId: string;
  artifactRefs: AutoSeedArtifactRef[];
  meta: Record<string, unknown>;
};

type V1TaskVersion = {
  task_id: string;
  active_step_key: string;
  artifacts_v1: { uri: string; kind_hint?: string }[];
  extras: Record<string, unknown>;
};

export function adaptV1TaskVersion(v1: V1TaskVersion): AutoSeedTaskVersion {
  return {
    id: v1.task_id,
    pipelineStepId: v1.active_step_key,
    artifactRefs: v1.artifacts_v1.map((a) => ({
      uri: a.uri,
      kind: a.kind_hint ?? "unknown",
    })),
    meta: v1.extras ?? {},
  };
}

In practice the real adapter handles nullability, optional nested blocks, and default values. The pattern is the same: isolate ugly field names and legacy nesting at the boundary so components never import V1* types.

Artifact and step registries

Seeding UIs need to answer two questions dozens of times per page: what kind of file is this, and which pipeline step does this identifier refer to? Hard-coded if (path.endsWith(".json")) branches rot quickly when new artifact classes appear or when multiple steps share similar filenames.

An artifact kind registry centralizes inference: it matches extension, MIME type, and path patterns in priority order and returns a discriminated union of kinds. A step registry resolves string step IDs to canonical step records, including disambiguation when names collide—think model_output_execution_0 style suffixes copied from legacy logs. Both registries use exhaustive switch on unions so TypeScript fails the build when a new kind or step arrives without handling.

That explicitness pays off when the thirteen-node pipeline adds a gate or renames a label: you update one table of metadata rather than chasing string literals across the tree.

Unified thirteen-node pipeline

The task lifecycle specification stopped being eleven flat steps. We modeled it as thirteen nodes: ten actionable steps plus three gates that represent approval or synchronization points in the lifecycle. Each node declares a structural type of either "step" or "gate" so the timeline can render different iconography and copy without string matching on identifiers.

All thirteen statuses share a single status registry: the authoritative map from status key to human label, icon name, semantic color token, and sort order for the timeline. Feature code imports that registry instead of scattering parallel constants in CSS and JSX. Task-level status for the overview is derived from currentStepStatus—there is no parallel status field that could disagree with the pipeline and confuse operators.

export const TASK_PIPELINE_STATUS_REGISTRY = {
  queued: {
    label: "Queued",
    icon: "clock",
    tone: "neutral",
    rank: 0,
    nodeType: "step",
  },
  awaiting_review_gate: {
    label: "Awaiting review",
    icon: "shield-check",
    tone: "warning",
    rank: 6,
    nodeType: "gate",
  },
  completed: {
    label: "Completed",
    icon: "check",
    tone: "success",
    rank: 12,
    nodeType: "step",
  },
} as const satisfies Record<
  string,
  { label: string; icon: string; tone: string; rank: number; nodeType: "step" | "gate" }
>;

export type TaskPipelineStatus = keyof typeof TASK_PIPELINE_STATUS_REGISTRY;

Gates are first-class nodes, not afterthought badges. That alignment between data model and UI makes the timeline honest: you see the same progression in the seeding platform that you saw in the desktop tool’s graph.

Task detail with desktop parity

The detail page used to fire three parallel fetches—manifest, task version, auxiliary metadata—and stitch results in the client. That increased failure modes and duplicated loading states. We collapsed the read path into one API response assembled on the server: run metadata banner at the top, step content routed by the current pipeline node, and a compact timeline that surfaces eight visible steps with overflow for the rest.

New viewers round out parity with the desktop tool. A model output panel pairs execution metadata (timing, model id, token counts where available) with an artifact browser rooted at the run’s output prefix. A golden data panel shows the human-improved reference content beside the same artifact affordances so reviewers can diff structure without leaving the browser. Labels and section order follow desktop terminology deliberately; the goal is continuity, not a redesign that forces a second training curve.

Keeping vocabulary aligned also reduces support load: search results, runbooks, and screenshots from the legacy tool still read as true when someone opens the same task in the browser.

Step content routing is a small map from pipeline node key to viewer component lazy-loaded per tab. Combined with the status registry, the page stays declarative: add a node in one place and both the timeline and the main panel learn about it.

Route template with versioned handlers

Next.js API routes stayed thin wrappers. A shared factory builds handlers for each API version so authentication, observability, and error envelopes match across endpoints. Inside the closure we resolve the injected TaskStorageProvider and call the right service.

type HandlerCtx = {
  provider: TaskStorageProvider;
};

export function createV1Handler<Body, Result>(opts: {
  methods: readonly ("GET" | "POST" | "PUT")[];
  run: (input: Body, ctx: HandlerCtx) => Promise<Result>;
}) {
  return async function handler(req: Request, ctx: HandlerCtx): Promise<Response> {
    if (!opts.methods.includes(req.method as (typeof opts.methods)[number])) {
      return Response.json({ error: "method_not_allowed" }, { status: 405 });
    }

    try {
      const body = req.method === "GET" ? (undefined as Body) : ((await req.json()) as Body);
      const result = await opts.run(body, ctx);
      return Response.json({ ok: true, data: result });
    } catch (e) {
      return Response.json({ ok: false, error: mapError(e) }, { status: 500 });
    }
  };
}

export const GET_TASK_VERSION_V1 = createV1Handler({
  methods: ["GET"],
  run: async (_body, { provider }) => {
    // parse taskId / version from URL, delegate to service
    return provider.getTaskVersion("taskId", "v1");
  },
});

When Prisma lands, the import that constructs ctx.provider changes; the handler factory and the service signatures do not.

What I would ship earlier next time

I would publish the provider interface as a tiny internal package or module the day the first manifest path is invented, even if only GCS exists. The psychological cost of “we will abstract later” is real, and the longer string paths live in handlers, the harder the extraction. I would also snapshot example manifests in tests as soon as the desktop tool exports them—fixtures beat oral tradition when debugging bucket ACLs at midnight.

Hot-swappable storage sounds like premature optimization until you watch a team ship a full seeding portal while migrations are still debating foreign keys. The provider-service split turned that schedule pressure into leverage: same product surface, same types, different backing store—but only because we disciplined the seams early and kept adapters at the edges.