Native observability — Everdict as the trace platform (OTel-first)

Status: DESIGN (maintainer decision, 2026-07-29). Supersedes-in-direction the edge-adapter half of execution-model.md §6: instead of "own the evidentiary copy, treat external platforms as import/export edges", Everdict owns the trace domain itself — its own OTel-standard ingestion, its own tenant-scoped trace store — and external platforms become egress mirrors plus import compat-shims. The §6 invariant ("never judge what you don't retain") and the data-plane triad stand unchanged; this design makes the invariant trivially true instead of carefully maintained.

Why (the maintainer's three arguments)

1. Everdict's agents must see traces. The agent is a first-class trace consumer — "analyze the last five runs", "why did this case fail" are trace queries. Pointing that capability at N external platforms means N query APIs, N auth flows, N schema drifts under the platform's core feature. A tenant-scoped trace view has to be Everdict's own.
2. Eval → operational runtime requires observability. If Everdict is to cover the operating of agentic services, not only their evaluation, then monitoring/observability of those services is table stakes. SigNoz proved the shape: embrace OTel as the ingestion standard, own the collector and the store, and you can stand up a new-level observability platform without owning any SDK.
3. Un-owned traces compound complexity. §6 documented five structural control losses (scores written to someone else's ledger, deep links born to dangle, best-effort export, register-time probe decay, an asymmetric sync ledger). Every new supported platform multiplies adapters (source + sink + probe + browse + inspect). The compromise in §6 was already regrettable — and the regret was signal.

The reframe that unlocks it

§6 held that "the external half is legitimate — the tenant's observability platform is where their organization lives, so absorption is impossible." That statement conflated two legitimacies:

• Legitimacy as a view — their org dashboards, their alerting, their habits. Real, and preserved: mirroring outward stays first-class.
• Legitimacy as the record — where the truth that our platform stands on lives. This was never legitimately theirs; we had just not built the alternative.

OTel dissolves the dilemma that made §6 settle for copies: ingestion is standardized. Owning the collector does not mean replacing their platform or shipping a proprietary SDK — instrumentation is a commodity (OTel SDKs everywhere; Langfuse/LangSmith/Phoenix SDKs and most agent frameworks can already emit OTLP), and the endpoint is a URL. The 037 lesson ("own the interface, reuse the standard") applies with a different minimum point of ownership: for images it was the auth plane; for traces it is the collector and the store.

What already exists (the foothold — this is a turn, not a build-from-zero)

Piece	Where	Today	Becomes
OTLP span parsing + GenAI-convention normalization	packages/trace/src/sources/otel.ts	pull mode (query their API)	the receiver's core, unchanged
everdict.run_id correlation tag	stamped at execution (application-execution/run-case.ts)	lets pull find our runs	native correlation on arrival
TraceProvenance extraction	sources/trace-source.ts	uniform "Everdict origin" across kinds	ingest-time provenance
Span waterfall + browse/inspect	spans-to-nodes, Settings › Traces	renders their store	renders our store first
TraceEvent vocabulary (+ raw span passthrough, artifacts)	contracts	the normalization target	unchanged — the internal contract → demoted to a read-time projection at N6 (otel-trace-model.md): TraceSpan becomes the record, TraceEvent[] stays exactly what graders and judges read
TrajectoryStore (design)	execution-model §6	the owned store for run trajectories	the same store, fed by the collector

The design

1. Ingestion front door — an OTel collector per install. OTLP/HTTP (+gRPC later) in. Tenant isolation at the door: per-tenant ingest tokens (mint/revoke — the token-server pattern from the managed image store, third appearance) route spans to the tenant's partition. v1 signal scope: traces; logs/metrics ride the same door on a later rung (N-O3).
2. The store — one port, a storage ladder. The TrajectoryStore grows into the tenant-scoped trace store behind one port: rung 1 Postgres-index + object-storage bodies (eval-scale; fits the compose stack as-is), rung 2 ClickHouse (ops-scale; the SigNoz-proven engine) as an opt-in adapter. Same isolation grammar as the filesystem (partition-per-tenant).
3. The contract — semantic conventions, not an SDK. Everdict defines its execution/eval vocabulary as OTel semantic conventions: everdict.run_id (exists), everdict.kind, everdict.case_id, everdict.group_id, plus the standard gen_ai.* conventions for LLM calls (model, tokens, cost). Scores/verdicts stay platform-layer records referencing trace ids — evaluation is our layer on traces, never span data someone else could overwrite.
4. Libraries for users, not adapters for us. everdict-otel packages (TS/Python): exporter config + semconv helpers + shims for popular agent frameworks; migration recipes for Langfuse/LangSmith SDK users (their SDKs speak OTLP — pointing OTEL_EXPORTER_OTLP_ENDPOINT at Everdict is config, not rewrite). This is the answer to "provide libraries usable from existing trace platforms".
5. Egress = collector fan-out. Mirroring raw traces to the tenant's platform becomes collector exporters (dual-write at the pipe — what collectors are for), replacing bespoke raw-trace sink code over time. The score-attach sinks (writing verdicts to their platform's API) remain — scores are ours and travel by our hand.
6. Imports demoted to compat shims. The pull sources stay for backfill and for platforms that cannot push — no longer the primary path. Two-phase traceRef collection dies naturally (the trace arrived live through the door); LiveTraceRef becomes an internal, stable link instead of one that documents its own decay.
7. The ops unlock — continuous evaluation. Production traces in the store + the native scoring path = judges over live traffic, platform events derived from trace facts, triggers waking agents on them. The observability market is moving observability→eval; Everdict attacks from eval→observability with the stronger eval engine.
8. Capacity plane hookup. Ingestion is a new admission lane in the §5 gate: per-tenant span/byte quotas, sampling policy, retention as a budget dimension. A trace firehose is the data-plane twin of runaway fan-out — same governance grammar (quota 429s at the door, never silent drops).

What it dissolves (§6's five losses, revisited)

Loss (§6)	After
attach-mode writes to their ledger	their trace is the mirror; ours is the record
deep links born to dangle	primary links are internal and stable; external links best-effort UX
export is best-effort	still is — and now that is fine, because a mirror may be lossy; the record may not
probe = register-time truth	no inbound dependency left to probe; egress mirrors keep their probe
asymmetric sync ledger	import provenance is native ingest metadata; export outcomes unchanged

Costs (honest)

• Storage engine + volume. Ops traces ≫ eval traces. The ClickHouse rung is real infrastructure (+1 compose container, operational learning). Hence the ladder: rung 1 default keeps self-hosted light.
• Retention becomes our bill. Per-tenant quotas, TTLs, sampling are product surface now, not config trivia. The usage meter grows an ingestion dimension.
• Collector operations. Backpressure, drop policy, availability. Self-hosted-first distribution softens this — each install runs its own door.
• Scope discipline. v1 = AI-execution observability (agents, LLM apps, services under test), not general APM. OTel carries metrics/logs too — later rungs by decision, not drift.
• No rip-out. Pull-mode features (MLflow live-verified, etc.) remain as shims; existing sink users keep working through the transition.

Phasing

Sequenced by execution-master-plan.md (plan of record — waves W0–W7; decisions locked at recommended values).

• N0 — Conventions + the door. Publish the everdict.* semconv; OTLP/HTTP receiver (api-embedded for v0) + tenant ingest tokens; normalize → TrajectoryStore. Dogfood: our own harness/agent traces land through the door (replacing two-phase pull where the harness can emit OTLP).

• N1 — Look inward. Settings › Traces primary tab reads our store (browse/inspect/waterfall reuse); LiveTraceRef → internal link; trace chips point home. First rung SHIPPED (master-plan W5): TrajectoryStore.list (metas only, keyset cursor) → GET /trajectories + list_trajectories (the browse surface over the owned ledger), Settings › Traces gains the PRIMARY "everdict traces" section (each row opens its run — the run is the home), and the run detail page dual-reads: row embed first, else the sealed trajectory (labeled with its provenance source) — this is how agent/sandbox/OTLP runs render a trace at all. The internal waterfall SHIPPED — and the reason it lagged was the MODEL, not the view: our ledger normalized every source into a flat TraceEvent stream whose only positional field was a scalar t, so the same trace was a span tree in the platform's own UI and a list in ours (an agent turn was worse still: t was a step index with no anchor, so there was not even a time axis). TraceEvent now carries optional STRUCTURE — spanId · parentId · durationMs · at — which normalization (spansToTraceEvents, parseOtlpSpans) and the agent's own transcript projection both fill in, and apps/web renders BOTH sources through one SpanWaterfall. Additive and optional throughout: a source that reports none produces exactly the stream it always did, and no judge/grader changes. Remaining: chat trace chips for own-store trajectories ride the existing run reference (zero new contract); the LiveTraceRef internal rewrite arrives with the N2 ingestion surfaces.

  Evidence is as private as the execution that made it. The ledger's browse surface published every member's agent-chat transcript to the whole workspace — a second door onto conversations the session store keeps owner-scoped. A sealed trajectory now carries an owner (mig 0116, backfilled from the run ledger); list filters on it in the QUERY (a page filtered afterwards would be short) and the detail read answers 404 for someone else's. The rule itself is runAudience in @everdict/domain — see docs/api.md §Run audience.

  And evidence has to be RECOGNIZABLE, or being retained is not the same as being there. Every browse row read <run id> · run · N events, so an agent conversation, an eval case, a sandbox shell and an OTLP arrival were the same row four times over: a member looking for the trace of the agent they had just run reported it as missing, and they were not wrong in any way that mattered. A trajectory now also carries kind (the run family) and label (the handle a person would use — the case id, the agent, the harness), denormalized at seal for exactly the reasons owner is: the ClickHouse rung has no run table beside it, and a name resolved after the page would either N+1 or lie. runEvidenceIdentity (@everdict/domain) is the one rule; mig 0124 backfills from the run ledger; list takes a kind filter in the same WHERE as the owner predicate, and Settings › Traces renders label + kind badge with a kind switcher.

  The handle names the PRODUCER; a row is told apart by its WORK. mig 0124's label fixed the four-rows- the-same case and left a fifth: for every kind but eval the label is the run's harness id, and for an agent run the harness IS the agent (Run.newAgentRun/newChatTurn), so one agent answering twenty questions seals twenty rows that all read default <uuid>. The label is a category — correct, constant, and useless as a handle. A trajectory therefore also carries preview (mig 0168): the one-line excerpt of what it was asked to do, derived from its own body by previewFromEvents (@everdict/domain) — the member's message, else the first tool call, else the root span, with the assembled invoke_agent <agent> root kept as the last resort because it names the producer again. It is stamped by NamingTrajectoryStore, a decorator wrapped once at the composition root: eight seal paths reach the ledger and only some hold a run record, so naming at the STORE also names the OTLP arrivals and materialized imports that no run-ledger join could ever reach. No backfill — the value lives inside the sealed body, and evidence is never rewritten. The web then leads each row with whichever of the pair actually varies (lib/row-text): the label for the work-named kinds, the preview for agent and for evidence that arrived unnamed, the other one as a chip, the id below both.

  The same defect, from the other end, on the platform lists. The external trace browser beside it drew name ?? "unnamed" + the trace id, where name is whatever the instrumentation called its root span — ChatCompletion on every trace in the project. Each platform already reports what its trace was asked to do; the adapters were reading the metrics and dropping it. TraceSummary now carries preview/userId/ sessionId and each adapter fills them from the response it ALREADY fetches (no extra call): MLflow's request_preview, Langfuse's input/userId/sessionId/metadata (which is where our own sink writes provenance — the list path was dropping the origin the inspect dialog showed), LangSmith's inputs (fetched for provenance keys, then discarded), a Phoenix root span's input.value, and — for Jaeger — the processes table, where the resource attributes including everdict.run_id live and which the adapter did not declare at all. See docs/architecture/trace-sink.md §F1 for the per-kind field table.

• N2 — Libraries + production ingestion. everdict-otel (TS/Py) + migration recipes; continuous evaluation (judges over live traces; platform events from trace facts). Continuous-evaluation rung SHIPPED (master-plan W6): the reserved trace source everdict points the pull machinery at the owned store — one-off (ingest/pull with source:{name:"everdict"}) and periodic (a pull-mode schedule with pull.source:"everdict" judges a rolling window each fire). No external platform, no re-upload, no materialize duplicate. Trace facts already flow (E4 thresholds). The library rung also SHIPPED: @everdict/otel (packages/otel, dependency-free semconv + exporter-config helpers, drift-guarded against the receiver) + docs/everdict-otel.md migration recipes (zero-code env, TS in-code, Python env-only — no Python package needed; the door's protobuf gap is documented with the collector front). Remaining: publish pipelines when the repo goes public.

• N3 — Scale rung. ClickHouse adapter; retention/quota surfaces; ingestion admission in the §5 gate. Governance half SHIPPED (master-plan W7): the ingestion admission lane — per-tenant events/hour quota at the door (workspace override via WorkspaceSettings.traceIngestion + GET/PUT /workspace/trace-ingestion, operator default EVERDICT_INGEST_MAX_EVENTS_PER_HOUR; the STORE is the meter via ingestedSince, so no counter drifts), refused at 429 with the arithmetic and announced as trace.ingestion_throttled (cooldown-bounded, trigger-matchable) — plus retention: EVERDICT_TRAJECTORY_RETENTION_DAYS drives an hourly deleteOlderThan sweep (logged, never silent; unset = keep forever). The ClickHouse adapter also SHIPPED (entered on the maintainer's measurement call): ClickHouseTrajectoryStore — the SAME port over ClickHouse's HTTP interface (SDK-free; parameterized queries, JSONEachRow inserts), swapped by EVERDICT_CLICKHOUSE_URL alone (ONLY the trajectory store moves; everything else keeps Postgres), with the opt-in --profile clickhouse compose service. Rung-2 honesties documented in the adapter: sealed_at as ISO String, first-write-wins resolved at READ (earliest row / argMin) over check-then-insert — and that clock read is now documented as the BEST-EFFORT answer it is. It resolves "which row sealed first" from a stamp each writer takes from its own clock, while WHICH attempt a case's verdict rests on was already decided in Postgres by the commit receipt; a duplicate carrying a backdated stamp therefore won the read and served the abandoned attempt's bytes under the run the receipt named. So the port carries an EXACT-IDENTITY read beside it — get(tenant, runId, { attemptId }) — which ranks the asked-for attempt above the clock and REFUSES a plane declaring a different one (a plane declaring none is kept: absence is not agreement, but it is not contradiction either). One rule, three impls: segmentDeclaresAttempt / trajectoryForAttempt (@everdict/application-control) for Postgres and in-memory, the same ranking stated in SQL for ClickHouse because there the duplicates are physical rows and they collapse before a caller could filter them (certified live — clickhouse-trajectory-store.scenario). ClickHouse's CREATE and its additive ALTERs are also derived from ONE column descriptor now, so a column cannot ship to fresh installs and be missing from the upgrade path. Remaining: byte quotas and sampling policy.

• N5 — The system plane (multi-emitter trajectories). A run is a SYSTEM, not one process: the agent under test, the orchestrator that placed it, and every service it drives. Before this rung the owned ledger held exactly one sealed body per run, so a service pushing its own OTel spans into a live run was rejected as a duplicate (rejectedSpans) and a topology run's late agent seal could lose to an early service arrival. SHIPPED: a trajectory is now the sum of its segments — one per emitter, keyed (runId, emitter) where the emitter is the arrival channel (run/otlp/import) or service:<service.name> (OTel's own attribute, the maintainer's decision — participating is OTEL_SERVICE_NAME plus the run correlation, never an everdict-specific key). Properties: first-write-wins is unchanged but now applies PER EMITTER, so "evidence is never rewritten" stays literally true while a late plane is added beside the others; each segment carries its own absolute t0, the anchor that lets planes share one time axis; and SealedTrajectory.events still resolves to the EXECUTION's own record (executionEmitter), so a service's spans can never displace what a judge scores. Storage: Postgres gains everdict_trajectory_segments (mig 0104, FK-cascaded, with a denormalized segment_event_count so a browse row and the ingestion meter stay honest); ClickHouse needs no new table — a segment IS a row, first-write-wins resolved at read per (run_id, emitter). The wire (GET /trajectories/:id, GET /runs/:id/trajectory, get_trajectory) gains segments, with the execution segment omitting events so a system read never ships the same trace twice. The web renders it as swimlanes (agent · placement · one lane per service) over one axis, and span.durationMs (new, optional) keeps a service's spans from arriving as instants. Remaining: W3C traceparent/baggage propagation helpers, and per-plane retention/sampling policy.

• N4 — Mirror consolidation. Collector-level exporters subsume raw-trace mirroring; score-attach sinks remain API-side. Blocked on N6 until now: a record with no trace id, no span kind/status, no span events and non-hex ids cannot be handed to an exporter as a valid OTLP trace.

• N6 — One model: spans are the record. See otel-trace-model.md. The N0–N5 rungs kept TraceEvent as both the storage record and the judge contract, which left two models for one thing and a lossy flatten at our OWN door (otlp-ingest-service parsed real OTLP spans and then called spansToTraceEvents before sealing). N6 restores OTel's upper layer: TraceSpan (OTLP-shaped, hex ids, kind/status/events/links, resource+scope) is the record; TraceEvent[] becomes a versioned read-time projection so graders/judges move not at all; kind:"infra" becomes a placement SPAN carrying the standard resource attributes (k8s.node.name/container.id) plus everdict.plane, so infra is an extension of the vocabulary rather than a second one; W3C traceparent propagation puts the whole system under one trace id, which demotes planes from a storage concept to a read-time group-by. Sealed TraceEvent[] bodies are never rewritten — the ledger records its body format per row. The agent RECORDS rather than reconstructs (TurnSpanRecorder off the kernel's onEvent), so a turn's latency, retries, fallbacks, compactions and subagents finally reach the evidence; the pull adapters' projection is folded into the one spansToEvents; and a scorecard stores the projection version its verdicts were read under.

Open decisions

• N-O1 — Storage default: rung 1 (PG+object) default with ClickHouse opt-in, or ClickHouse from N0? Rec: rung 1 — the compose stack stays light; the port makes the swap invisible.
• N-O2 — Receiver placement: embedded OTLP/HTTP endpoint in the api (N0) vs sidecar stock collector. Rec: embedded first, sidecar at N2 for gRPC/perf/fan-out.
• N-O3 — Signals: traces-only v1; logs/metrics later? Rec: traces-only — scope discipline.
• N-O4 — Do bespoke raw-trace sinks retire at N4? Rec: yes, after collector exporters prove parity; score-attach sinks stay.
• N-O5 — Sampling defaults: head vs tail sampling and who owns the knob (platform default + tenant override). Rec: no sampling at eval-scale; tail-sampling introduced with the ClickHouse rung.

Non-goals

• Not general-purpose APM in v1 — no infra-metrics dashboards; AI-execution observability first.
• Not replacing the tenant's org-wide observability — mirrors are first-class and the collector fan-out exists precisely to keep their dashboards fed.
• Not a proprietary SDK — conventions over standard OTel only; leaving Everdict is removing an endpoint, which is exactly why choosing it is safe.

Cross-links: execution-model.md §5–§6 · trace-sink.md (raw-mirror half superseded at N4; score-attach half stands) · workspace-scoped-integrations.md (trace-source pool → shim registry).