Long-Horizon RL Task Pipelines: From Scenario Design to Golden Data

Most AI training datasets are built from short tasks: classify a sentence, fix a function, rate a response. Long-horizon reinforcement learning needs something harder — occupational scenarios that take a human expert hours of realistic effort, produce multi-artifact deliverables, and survive a multi-gate review before they become golden data.

On the enterprise evaluation platform, I spent roughly four months shipping the auto-seed pipeline that produces those tasks — claim lifecycle, events table, LLM dedup adjudication, trainer-gate QC, taxonomy search, configurable timeouts, user dashboard, and 100+ merged PRs around them. This post is the architecture view: what "long-horizon" means in practice, why a multi-gate human pipeline beats single-shot annotation, and which edge cases only appear when work spans hours instead of minutes.

Long-horizon RL tasks fail without multi-role claim pipelines, taxonomy, and QC gates — not just better prompts.

What "long-horizon" means in RL task design

A long-horizon task on the platform is identified by an lh-* ID — for example lh-government-compliance_officers-20260529-231730. The ID encodes sector, role, and a timestamp. The content is not a coding puzzle. It is a multi-step occupational scenario:

• A compliance officer reviews a policy packet, drafts findings, and packages evidence
• A financial analyst reconciles statements, builds a memo, and cites source artifacts
• A legal researcher synthesizes case materials into a structured recommendation

These scenarios demand hours of realistic expert effort. Deliverables are multi-artifact — documents, tables, checklists, Harbor-style packages — not a single string label. The model being trained or evaluated must plan, use tools, revise, and persist state across a long trajectory. If your annotation UI assumes a five-minute turnaround, the data you collect will look like short tasks no matter how carefully you write the prompt.

That mismatch is why the auto-seed pipeline exists. Scenario design, model runs, golden solutions, calibration, and final review are separate stages with separate human owners. The pipeline is the coordination layer that makes hour-scale work producible at volume.

The ~11-step pipeline

A task moves through roughly eleven stages from design to finished golden review:

1. task_design — trainers author the scenario and acceptance criteria
2. … intermediate packaging, model run, and intake stages …
3. trainer_gate — trainers review and approve the designed task
4. model_output_review — experts examine model trajectories against the scenario
5. golden_data — experts produce the golden solution
6. calibration — golden solutions are calibrated for reward / eval use
7. golden_review — final human gate before the task is Finished

Horizon span	Stages	Claim role	Job
A1 — Authoring	task_design → trainer_gate	AUTHOR	Design and review the scenario
A2 — Golden data	model_output_review → calibration	GOLDEN_DATA	Produce golden solutions vs model output
A3 — Final review	golden_review	REVIEWER	Final quality gate

Each stage has its own claim, timeout, release semantics, and admin kill-switch. Diagrams that collapse this into "human labels data" hide the actual product: a state machine that keeps multi-hour work from colliding, stalling, or shipping incomplete golden forms.

Why a multi-gate pipeline beats single-shot annotation

Single-shot annotation works when the unit of work is small and the annotator's mistake is cheap. Long-horizon occupational scenarios fail that test in three ways:

1. Expertise is split. The person who designs a government-compliance scenario is rarely the same person who should write the golden solution against a model trajectory — and neither should be the final reviewer of their own work.
2. Failure modes accumulate. A vague scenario produces an unusable model run; an incomplete golden form produces a useless reward signal; a skipped QC check ships both.
3. Time makes exclusivity necessary. If one person can hold a task for hours, you need claim timeouts, release, and reclaim — not a shared Google Doc.

Multi-gate pipelines look bureaucratic until you measure what single-shot produces: scenarios that look complete in a UI but cannot grade a long trajectory, or golden data that mirrors the author's bias because the same human designed and "solved" the task.

Claim roles as the coordination mechanism

The core abstraction is not a ticket status — it is a claim role spanning a horizon of stages:

• AUTHOR (A1) — claims from task_design through trainer_gate
• GOLDEN_DATA (A2) — claims from model_output_review through calibration
• REVIEWER (A3) — claims golden_review

Different humans must hold different roles on the same task. That rule is enforced at claim time, not as a soft guideline in a wiki:

async function claimTask(
  taskId: string,
  userId: string,
  role: ClaimRole,
): Promise<ClaimResult> {
  const prior = await claims.forTask(taskId);

  // Same human cannot hold AUTHOR and GOLDEN_DATA on one task
  if (prior.some(c => c.userId === userId && c.role !== role)) {
    throw new ClaimConflictError(
      'Different claim roles on the same task require different humans',
    );
  }

  if (await killSwitch.isGateDisabled(role.gate)) {
    throw new GateDisabledError(role.gate);
  }

  return claims.create({
    taskId,
    userId,
    role,
    expiresAt: Date.now() + timeouts.forRole(role),
  });
}

Claim timeouts, manual release, and admin kill-switches are per gate. AUTHOR work can time out independently of GOLDEN_DATA work. An admin can freeze trainer_gate without freezing golden_review. That granularity is what turns "people will figure it out" into a production system.

Taxonomy + scenario as the front door to diversity

Long-horizon tasks are only useful if they cover real occupational diversity. The platform's RL-World taxonomy is an extensible L(n) tree:

Sector → Sub Vertical → Function → Role

Taxonomy nodes are seeded from YAML into GCS, then consumed by cascading bulk intake. Required scenarios hang off roles so intake cannot create a task without a grounded occupational context. Search across the taxonomy is a first-class product surface — trainers pick a role, not a free-text job title that later collapses into noise.

type TaxonomyNode = {
  level: 'sector' | 'sub_vertical' | 'function' | 'role';
  id: string;
  label: string;
  children?: TaxonomyNode[];
  requiredScenarios?: ScenarioSpec[];
};

async function bulkIntake(roleId: string, count: number): Promise<TaskDraft[]> {
  const role = await taxonomy.getRole(roleId); // GCS-backed, YAML-seeded
  const scenarios = role.requiredScenarios;
  if (!scenarios?.length) {
    throw new IntakeError('Role has no required scenarios');
  }
  return scenarios.flatMap(s => seedTasks(role, s, count));
}

Without taxonomy as the front door, "diversity" becomes whatever trainers typed last week. With it, auto-seed can cascade intake across sectors while keeping scenario requirements attached to the leaf roles that actually do the work.

Golden data + QC as the quality floor

Golden data is the quality floor for RL rewards and evals. If the golden solution is incomplete, every downstream score is fiction. Auto-QC agents run at trainer gate and golden-data stages via Cloud Tasks workers with webhook callbacks — the same request-async-respond pattern used elsewhere on the platform:

1. Human submits at a gate → manifest transitions to submitted / qc_pending
2. Cloud Tasks worker dispatches configured QC agents
3. Webhook callbacks return results
4. Pass advances the stage; fail returns the task with actionable findings

Force-pass is an admin escape hatch for known false positives. It is blocked when the golden form is incomplete. That guard sounds small; it is the difference between "QC can be overridden carefully" and "admins can ship empty golden data under pressure."

async function forcePassQc(
  taskId: string,
  adminId: string,
): Promise<void> {
  const golden = await goldenForms.get(taskId);
  if (!golden || !golden.isComplete()) {
    throw new ValidationError(
      'Force-pass blocked: golden data form is incomplete',
    );
  }
  await qc.forcePass(taskId, { adminId, reason: 'admin_override' });
  await events.record('qc.force_passed', { taskId, adminId });
}

Stack underneath: Next.js UI, PostgreSQL + Prisma for claims and manifests, Zod for payload validation, GCS for taxonomy and packaged artifacts, Claude / GPT for adjudication and dedup where LLM judgment beats brittle rules. Public tech is enough to describe the system; the hard part is the coordination semantics, not the model names.

Edge cases that only show up at horizon scale

Minute-scale annotation hides failure modes that dominate hour-scale work. These are the ones that consumed real engineering time:

Stale claims

A trainer claims AUTHOR work, starts a draft, then disappears for six hours. Without timeout + release, the task is a ghost lock. With timeout, you still need draft cleanup and manifest reset to pending so another AUTHOR can reclaim without inheriting a half-written, ownership-ambiguous draft.

Release and timeout reset

Manual release and auto-timeout must share the same reset path: delete in-progress draft, return stage to pending, preserve audit history. Divergent paths produce "I released it but it still looks claimed" bugs that destroy trust in My Tasks.

async function onClaimRelease(
  taskId: string,
  reason: 'manual' | 'timeout',
  role: ClaimRole,
): Promise<void> {
  await drafts.delete(taskId, role);
  await manifest.transition(role.gate, 'pending', {
    releasedAt: Date.now(),
    releaseReason: reason,
  });
  await events.record('claim.released', { taskId, role, reason });
}

Spinner scoping

Status changes fire constantly — claim heartbeats, webhook callbacks, draft autosave. If the Auto-QC spinner shows on every transition, reviewers stop believing any spinner. Scope progress UI to "this gate is submitted AND QC job is in_progress." Anything else is noise.

Force-pass guards

At horizon scale, incomplete golden forms are common mid-flight. Blocking force-pass until the form is complete prevents the exact failure mode pressure creates: skip QC, ship now, fix never.

Edge case	Symptom at horizon scale	Fix
Stale claim	Task locked for hours with no progress	Per-role timeout + reclaimable pending
Release / timeout reset	Ghost drafts, wrong My Tasks state	Shared reset path + events audit
Unscoped spinner	Reviewers ignore all progress UI	Spinner only during active QC
Force-pass abuse	Empty golden data enters eval corpus	Block when golden form incomplete
Same-human dual role	Author grades own scenario	Enforce role exclusivity at claim

Scale of the engineering

None of this landed as one epic. Over ~four months the surface area included:

• Claim lifecycle across AUTHOR / GOLDEN_DATA / REVIEWER
• Events table for pipeline observability
• LLM dedup adjudication for near-duplicate scenarios
• Trainer-gate and golden-data Auto-QC wiring
• Configurable claim timeouts and admin kill-switches per gate
• Taxonomy search and YAML → GCS seeding
• User dashboard for workload visibility

100+ merged PRs is the real signal: long-horizon pipelines are not a prompt engineering problem. They are a product of claim semantics, taxonomy intake, and QC placement — shipped incrementally against production edge cases.

What I'd do differently

I would publish the three claim roles and their stage spans as a formal state machine on day one, before the first trainer-gate PR. We discovered exclusivity rules and timeout interactions through production tickets; a diagram in the repo would have caught several of them in design review.

I would also treat golden-form completeness as a hard invariant earlier. Force-pass guards arrived after incomplete golden data was already a known risk. Invariants that protect reward quality should ship with the first golden-data gate, not as a follow-up.

The pattern: coordinate the horizon, don't just annotate it

Long-horizon RL needs tasks that take humans hours, not minutes. That requirement forces a multi-gate pipeline, multi-role claims, taxonomy-backed scenario intake, and QC at the gates where humans can still fix failures. Better prompts help inside a stage. They do not replace the coordination machinery between stages.

If you are building evaluation or training data for agents that plan over long trajectories, start with claim roles, taxonomy, and QC placement. The golden data you get will be only as good as the pipeline that refused to ship incomplete work.