Skip to main content

Everdict docs

Every document in this tree is listed here. Conventions (single source of truth): CLAUDE.md + ../.claude/ (rules + skills).

Two audiences, one tree. guide/ is the product documentation — written for someone using Everdict, and the tree the public docs site publishes. Everything else here is maintainer-facing: reference pages, design records, runbooks. See architecture/docs-site.md for how the two map onto the site.

Guide — product documentation​

Start here (maintainers)​

Surfaces​

  • api.md — the control-plane HTTP API (apps/api): runs, scorecards, datasets, judges, runtimes, schedules, bundles, workspace integrations, runners — async POST /runs + poll/webhook
  • mcp.md — the agent-facing MCP server (/mcp): OAuth-protected (Keycloak) + API keys, role-gated, full BFF↔MCP parity
  • web.md — the SaaS web (apps/web, Next.js FSD, Linear-style): Keycloak login, /{workspace}/… dashboard, workspace settings, personal account page
  • architecture/desktop-app.md — the desktop app (apps/desktop, Electron): web-parity shell + resident self-hosted runner + one-click pairing + auto-update + 3-OS release CI
  • architecture/one-call-sdk.md — reproduce env + N trials + score → verdict in one await
  • everdict-otel.md — @everdict/otel: sending traces to Everdict (migration recipes)

Eval entities​

  • registry.md — versioned SSOT (@everdict/registry): harnesses · datasets · judges · runtimes, (tenant, id, version), immutable versions, _shared fallback
  • datasets.md — harness-agnostic eval-case bundles (import, provenance, recipes)
  • judges.md — Agent Judges: model (LLM/VLM call) | harness (delegate an agent), applied per-trace on scorecards
  • models.md — workspace-registered LLM models (provider · model · baseUrl · apiKeySecret), referenced by id from a judge/harness
  • runtimes.md — tenant-registered execution infra (local | nomad | k8s); "my machine" → self-hosted runner
  • scorecards.md — batch evals (dataset×harness → Scorecard+summary), baseline↔candidate diff, push/pull trace ingest, leaderboard
  • suites.md — suites & version regression (everdict suite, scorecard diff)
  • command-harness.md — declarative command harness: bring any CLI agent as a HarnessSpec, no code adapter
  • service-harness.md — service-topology harnesses (multi-service + browser/OS target env), Nomad/K8s, OTel/MLflow trace
  • architecture/harness-taxonomy.md — Template (category) + Instance, pins and resolution
  • architecture/harness-playground.md — interactive test cases against a live harness session
  • architecture/eval-domain-model.md — the Dataset / Rubric / Grader split
  • architecture/standard-task-formats.md — bring an existing agent benchmark, run it managed
  • architecture/bundles.md — one-shot self-serve registration (harness + benchmark + runtime as a unit)
  • architecture/managed-case-image.md — the case.image agent-bootstrap contract

Scoring & verdicts​

Execution & placement​

Self-hosted runners​

Environments & images​

Observability​

Work, knowledge & the product axis​

Agents​

Tenancy, auth & integrations​

Billing & metrics​

Database migrations​

The public docs site​

Internals — historical design records​

Written during the re-architecture review and not maintained since. The umbrella migration SHIPPED on 2026-07-10, so the packages/{core,suite,run-case,billing} and apps/api/src/core/** paths they cite name the pre-migration layout. Read them for the reasoning, not the addresses.