Skip to main content

Orchestration (durable control plane)

Role charter vs the event plane: see architecture/event-plumbing.md — one line separates the two ("does the work have a definition of done?"): Temporal owns completion-bearing plans (batch driver, schedule clock) and stays optional (Direct fallback); the event plane owns open-ended perception and reaction. Temporal fires facts and executes plans; it never routes facts or decides reactions.

Above routing, the control plane can run each case as a durable Temporal workflow so runs survive control-plane restarts and retry transient backend failures.

Why a workflow engine at all — the DIY ladder​

What Temporal sells is not "workflows"; it is a durable program counter. In an ordinary program, "where was I" — the loop index, retry counts, partial results, what comes next — lives in RAM and dies with the process. The work itself may survive (cases keep running on runtimes); knowing where you were does not. Our own code states the problem verbatim (runStartupRecovery): "batches/runs are tracked in-process … at restart any queued/running record is a ghost with no one to resume it."

The ladder you climb without an engine (we are standing on rung 1):

  • Rung 0 — in-process loop (DirectOrchestrator): restart = ghosts. Fine for dev.
  • Rung 1 — checkpoint in the DB + a recovery sweep (what we built: the run ledger as checkpoint, recoverInterrupted, adoption-then-caseSpec-re-dispatch (mig 0051 exists for this), tombstones, background adoption so a long run doesn't block startup). The tax: the recovery path is a second implementation of the forward path and must agree with it forever — every feature (trials, spillover, retry classes, streaming phases) lands twice. And four things are still missing: durable timers ("retry in 90s", "wait 3 days" ⇒ next_*_at columns + a poller = a scheduler reinvented in a table), HA safety (two replicas both sweep ⇒ hand-rolled leases), in-flight versioning (deploy during a 2-hour batch: the checkpoint says plan A, the new code is plan B), and signal delivery (cancel must find the right process mid-await).
  • Rung 2 — generalize it for the second and third durable process (approval waits, reapers, retention): process table + step executor + retry policies + timer poller + leases + idempotency + versioning + a debug surface… you have written a workflow engine, minus the history/replay debugging, the query language, the batch operations, and the gRPC surface the ops agent needs (044).

What the engine mechanically provides: deterministic replay (any worker rebuilds RAM — locals and program counter — from the event history and continues at the next line; the forward path IS the recovery path, one implementation), server-owned durable timers (sleep(72h) fires even if every worker was down), declarative per-activity retry policies, workflowId dedup, durable signal delivery, task-queue HA without leases, patch APIs for in-flight versioning — and the history that powers the Driver ops surface for free.

What it does NOT remove (honesty): effect idempotency stays ours. If a worker dies mid-activity the activity retries — dispatchCase must not double-dispatch, which is exactly why the adoption/zero-re-run discipline remains valuable with Temporal. The engine erases "where was I" (control state), never "is it safe to do twice" (effects). And for open-ended convergence (autoscaler, event consumers, anything level-triggered), a reconciliation loop over desired state is the better tool — the charter's line holds from the other side.

Break-even, stated plainly: one durable process with short steps → rung 1 is a defensible price (we paid it). Days-long waits, per-step retry policies, HA replicas, deploys during long runs, and — above all — a growing count of distinct durable processes (Tier 1 alone adds four) → the per-process tax exceeds the cost of the engine. Keeping Temporal is the decision to stop paying that tax per process; the size of runStartupRecovery is the running meter of what rung 1 costs.

Workflow catalog — charter-filtered expansion (DESIGN)​

The ops-agent audit (event-plumbing.md, The ops-agent test) settled that Temporal stays — which raises the follow-up: use it better. Every candidate below passed the charter filter (definition of done + must survive crashes/time + not open-ended); everything that failed is listed in the anti-catalog so it cannot creep in later.

Running today (6): evalCaseWorkflow · suiteWorkflow · scorecardBatchWorkflow (+ the workflow-owned retry batch) · scheduledScorecardWorkflow + TemporalScheduleDriver (the clock) · scoreGroupWorkflow (everdict-score-<groupId> — Tier-1 item 3, SHIPPED in W2: the detached phase-2 pass; prepareScore's once-per-pass strip-first [the prepared flag rides continue-as-new] makes the id-only measured predicate mean "judged in THIS pass", then planScore's unfinished-only idempotence + scoreGroupCase's skip-if-judged give restart-safe, zero-duplicate re-scoring; start failure degrades to the in-process pass. The loop is PLAN → EXECUTE → REPLAN (arch-review 15 P1-6): it used to plan once and then choose between finalize and rotate on plan.keys.length > limit, so a case's right to a retry depended on how many OTHER cases sat beside it — 601 cases overflowed the slice and re-planned, 2 cases finalized after a single attempt, and MAX_JUDGE_ATTEMPTS_PER_PASS described only the larger batch. A pass now finishes when the worklist is EMPTY, which only a fresh plan can answer. Termination is the LOOP's own (decideScoreRound, a pure decision unit-tested without a Temporal test environment): a case skipped by the judge stream leaves no row at all, so its attempt budget never engages and "replan until empty" would bill forever — the stall guard bounds it at MAX_STALLED_SCORE_ROUNDS, derived to dominate the retry budget rather than picked, and its count rides continue-as-new so rotating cannot launder a stuck pass into a fresh budget. Giving up is RECORDED (finalizeScore(abandoned)), because "we stopped retrying" is not "there was nothing left to do". The judgment claim's ordinal is the LOGICAL ROUND, not the rotation count (arch-review 16 P0-1): every round schedules a new activity execution and Temporal's attempt restarts at 1 in each, so an ordinal that only moved on continue-as-new made a round's first attempt lose to the previous round's exhausted ones — the case could then never finish, cancelling the replan loop's purpose for exactly the retry it exists to serve. Rotation CARRIES the round; the execute decision advances it) · approvalWorkflow (everdict-approval-<id> — Tier-1 item 1, SHIPPED in W2: the durable approval WAIT; decision signal or days-long timer → deny-on-expiry via the internal bridge, idempotent against a settled record; the agent loop stays in the agent service — the workflow owns ONLY the wait).

Tier 1 — adopt next (each unlocks a roadmap phase):

  1. Durable parked approvals — approvalWorkflow(approvalId): park → notify (activity) → durable wait for the decision signal or a days-long timer → resume/deny. Done = approved | denied | expired. Replaces the in-process park (10-min deny-on-expiry; an agent-service restart expires as deny — agent-automation's recorded v1 bound). Unlocks: A6's full shape; default permission mode becomes usable for headless automation. The agent loop stays in apps/agent — the workflow owns only the WAIT. SHIPPED (W2) — see "Running today"; the resume leg (a post-restart decision re-drives the run as a continuation turn from its transcript) lives in the agent service, exactly as chartered.
  2. Session/warm reapers as durable per-entity timers — reaperWorkflow(runId): sleep(ttl) + extend/close signals → teardown activity. Done = torn down. Unlocks: execution-model P6 ("the reaper is the finally" made crash-proof for sandbox runs); browser sessions fold in later (O6); warm-pool idle teardown is the same pattern. SHIPPED (W5) as sessionReaperWorkflow (everdict-reaper-<runId>): started at sandbox create, closed signal on close (prompt no-op), deadline → reapSession activity → POST /internal/sandboxes/:id/reap — a live handle expires normally; a handle lost to a CP death settles the row as orphaned and removes the stray container by the row's session.computeId (Driver.reap). The reap activity retries UNBOUNDED (capped backoff): a CP outage is exactly the case the reaper exists for, so it never gives up while one is down. Ops family reaper (ledger id = the run id) on /ops/driver. The ledger orphan sweep is the safety net under all of it: sweepOrphans (boot + interval, sandbox and browser lanes alike) reaps any running session row past its deadline + grace whose handle no live process holds — so a reaper that never armed (startReaper is best-effort), a wiped namespace, or a row written by a process with no reaper wired can no longer leave a permanent running zombie. Boot recovery correspondingly LEAVES session-kind runs alone (they are not resumable work — no caseSpec, no backend job), and a background resume that can neither adopt nor re-dispatch now tombstones the run it claimed (INTERRUPTED) instead of silently abandoning it.
  3. Phase-2 scoring — scoreWorkflow(groupId, spec): judge N×M with per-case retries → aggregate → persist. Done = scored + aggregated. Unlocks: execution-model P2 — re-scoring a 500-case group survives restarts instead of dying with the process. SHIPPED (W2) as scoreGroupWorkflow — see "Running today" above; only runIds-backed groups route to it (an embed group has no per-case store for idempotent write-back, so it takes the in-process pass).
  4. Heavy event reactions (the E3 executor) — a thin consumer starts workflowId = eventId (idempotent by construction); the workflow runs the multi-step reaction. First residents: regression triage, the scorecard-fix-PR chain. SHIPPED (W6 backlog close-out) as reactionWorkflow (everdict-reaction-<eventId>-<subscriptionId> — the id includes the rule, so two subscriptions on one fact chain independently while redelivery still collapses): the subscriptions:reactions cursor consumer starts it for reaction.kind="workflow" subscriptions; each step = one agent activation over the activity → CP → agent-service internal bridge, watched to terminal under a per-step budget (awaiting_approval counts as alive — a HITL park mid-chain is exactly what durability buys). A step that ends failed/cancelled/skipped/timed out ends the chain; the workflow adds no judgment — the runs already narrate on the log. Ops family reaction (ledger id = <eventId>-<subscriptionId>).

Tier 2 — adopt opportunistically: cascade-cancel walker (O8: done = every non-terminal descendant cancelled; big trees need retries against runtimes. O8's per-kind OPT-OUT lands here too, deliberately: today's walk revokes exactly one descendant species — caused scorecard batches — so a kind knob would have nothing to discriminate; the walker is what brings run-level descendants [standalone-run cancel machinery, caused sandbox sessions] and with them the knob's first real choice) · pull-ingest shim pipeline (pull → materialize → judge → attach-back; the demoted-but-alive import path) · retention/TTL sweeps on Temporal Schedules (image-store GC; the event-log TTL and trajectory retention shipped their FIRST rung as in-process hourly sweeps — EVERDICT_EVENT_RETENTION_DAYS / EVERDICT_TRAJECTORY_RETENTION_DAYS, unset = keep forever — the Temporal-Schedules rung takes over when multi-replica CPs make in-process timers unreliable) · CI re-pin (merge → digest resolution → new immutable version; small but flaky-prone).

Tier 3 — hold: export attach-back durability (the charter allows a mirror to be lossy — only if attach failures become a real burden) · benchmark/dataset imports (one-shot; only if size demands). Boot recovery should SHRINK, not grow: every Tier-1/2 adoption moves state into workflows that recover themselves; runStartupRecovery remains only for what workflows don't own.

Anti-catalog (charter-failed — never workflows): event routing/fan-out (rule 1) · the agent loop itself (not a second agent runtime) · autoscaler/scheduler control loops (continuous, no completion) · notification fan-out (a cursor consumer) · the admission gate · a session's interactive I/O (only its TTL timer).

Expansion disciplines (what keeps "more Temporal" safe):

  • Deterministic workflow IDs are the correlation grammar — workflowIdFor(scorecardId) exists; extend the family (approval:<id>, reaper:<runId>, score:<groupId>, reaction:<eventId>) so idempotency and ledger-vocabulary addressing come free.
  • Every new workflow lands in the Driver ops surface on day one — describe/pending/cancel visible to the ops agent; a workflow the agent can't see violates the adoption gate that justified Temporal. v0 SHIPPED (W2): GET /ops/driver/:family/:id + POST …/cancel (families batch|score) ↔ MCP describe_driver_workflow/cancel_driver_workflow — ledger-vocabulary addressing (a scorecard/group id, never a raw workflowId), ledger-ownership scoping, read = runtimes:read, cancel = runtimes:control. approval:/reaper:/reaction: join the family enum as their waves land. Force-termination + the operator inventory (zombie control): POST /ops/driver/:family/:id/terminate ↔ MCP terminate_driver_workflow (same ownership scoping/gates as cancel) stops the workflow cancel cannot reach; the OPERATOR plane (x-internal-token, deliberately outside workspace scoping because a leaked workflow's ledger record is gone by definition) gets GET /internal/driver/workflows (every everdict-* workflow across tenants, family/ledger parsed back out of the id grammar) + POST /internal/driver/workflows/terminate (raw workflowId; refuses non-everdict workflows). Neither writes the ledger — the recovery sweeps settle whatever row a dead workflow owned. The schedule clock self-heals too: a fire whose record is gone deletes the orphan Temporal schedule at the fire choke point, TemporalScheduleDriver.remove no longer swallows real failures (the DB row survives a failed driver delete), and boot runs ScheduleService.reconcile() (Temporal⟶DB, DB is SSOT).
  • Workflows never emit facts directly — their activities' state transitions do (same-tx outbox), so the event plane's "transition ⇒ fact" invariant holds inside workflows too.
  • Activities that create demand pass the §5 gate — a workflow is not a side door around admission.

Two orchestrators (@everdict/orchestrator)​

  • DirectOrchestrator(dispatcher) — runs in-process via a Dispatcher (Router or Scheduler). Simple; dies with the process.
  • TemporalOrchestrator({address, taskQueue}) — client: starts a workflow and awaits its result.

Topology (dispatched worker + Temporal)​

everdict run --orchestrator temporal  (client)  ── start workflow ──▶ Temporal Server
                                                                        │  task queue
everdict worker  (long-running)  ◀── poll ─────────────────────────────────┘
   holds Scheduler(registry) → activity dispatchCase(job) = Scheduler.dispatch → Backend → agent → CaseResult
  • Workflow (evalCaseWorkflow / suiteWorkflow) is deterministic — it only calls the dispatchCase activity (retry + 1h start-to-close timeout). No I/O in the workflow. suiteWorkflow uses a bounded lane count so a big suite can't flood activity slots.
  • Activity dispatchCase does the real backend dispatch via a Dispatcher. The worker wires a capacity-aware Scheduler (gates on Backend.capacity(), queues when full) — see docs/execution-backends.md.
  • The worker holds the BackendRegistry + Scheduler; the client (CLI) just starts + awaits.

Run it (self-hosted dev)​

# 1) Temporal dev server (gRPC 7233, UI http://localhost:8233)
docker compose -f deploy/temporal/docker-compose.yaml up -d

# 2) worker — holds the backends (here: default single local backend)
pnpm everdict worker --temporal-address localhost:7233
#   multi-cluster: pnpm everdict worker --backends-config backends.config.json

# 3) client — durable run (blocks until the workflow completes)
pnpm everdict run --orchestrator temporal --task "..." --test "..."

Jobs route by placement.target (set via --target); suites fan out via suiteWorkflow.

Default --orchestrator direct keeps the in-process path (no Temporal needed). Production: use a persistent Temporal deployment (auto-setup + Postgres/Cassandra).