Open-source agent evaluation runtime

Know if your agents
actually work.

You changed a prompt, swapped a model, upgraded a framework. Did it get better? Everdict runs any agent against your data, scores every version the same way, and gives you a verdict with the trace behind it.

Get started How it works

Apache-2.0 · no vendor sandbox Self-hosted · your keys, your clusters Harness-agnostic · no rewrite

first-scorecard

# bring the stack up, then run a real evaluation
$ docker compose -f deploy/compose/docker-compose.dev.yaml up -d
$ bash examples/quickstart/run.sh

  ① registering harness   demo-agent@1.0.0
  ② registering dataset   demo-smoke@1.0.0
  ③ running scorecard     sc_91f2ab
  ④ verdict               2/2 passed  passRate 1

# the only question that matters
$ curl '.../scorecards/diff?baseline=sc_7c01&candidate=sc_91f2ab'

  regressed  retrieval/long-context   pass → fail
  improved   retrieval/citations      fail → pass

Why

Eyeballing outputs does not survive a disagreement.

The usual answer to "is it better?" is that somebody ran it on a few examples and read the result. That cannot gate a pull request, cannot catch a regression three weeks later, and cannot be shown to anyone who disagrees.

MEASURE

Score every version

Run a fixed dataset, score every case the same way, diff baseline ↔ candidate. Regressions get named case by case, not averaged away.

ANY AGENT

Evaluate what you already built

Claude Code, Codex, a CLI you wrote, or a multi-service stack. A harness is a JSON document, not an adapter you have to write.

YOUR INFRA

Nothing leaves your network

Runs land on your Nomad, your Kubernetes, or your laptop. Bring your own models and keys. Air-gap capable.

The part most tools get wrong

A verdict has three states, not two.

When a grader throws or a judge key expires, that case did not pass — and it did not fail either. Collapsing "not evaluated" into either one is how eval tooling produces confident nonsense.

passed

Evaluated, and it met the policy stamped onto this batch at submit time.

failed

Evaluated, and it did not. The trace behind the number is still there.

not evaluated

Nothing measured it. Never counted as green — by the gate, the chart, or the release.

A batch of 100 cases whose judge key expired halfway reports 0.82 over the 50 measured — not 0.41 over 100. Every invariant like this one is pinned by a nightly suite that prints PASS or the name of what broke.

How it works

Four moving parts, kept apart on purpose.

Grading lives outside the harness, so two different agents solving the same case are scored by the same code. That separation is the only reason their numbers can be compared at all.

Register a harness

The agent under test, declared as JSON. Drive it over a process boundary — no SDK, no rewrite.

Point it at a dataset

Cases describe the task and the world, never the agent. That is what makes them portable.

Run a scorecard

Every case, N trials each, placed on the runtime you chose. Asynchronous, resumable, durable.

Diff against the baseline

Named regressions and improvements — the call a CI gate makes to block a pull request.

One run: replay player, the judge's verdict with its written reasoning, per-metric scores, and the full trace

One run — replay, the judge's reasoning, and the trace behind every number.

Batch evaluation

Dataset × harness, and the trend that follows.

Scorecards seal the exact versions they evaluated, so a comparison made next quarter still means what it meant today.

Everdict scorecards — dataset × harness runs with judge scores, models, and pass rates

Who drives it

People in the browser. Agents over MCP.

Both surfaces expose the same capabilities, kept at parity by construction — a tool exists on both or on neither.

HUMANS

The web app

Workspace dashboard, scorecard analysis, saved views, and the eval tracker that says why you are evaluating in the first place.

AGENTS

MCP + API keys

Two commands add Everdict to a Claude Code session — the tools and the domain context, so it knows what a harness is before it calls one.

GitHub Actions

Keyless OIDC. Evaluate a pull request, re-evaluate on a comment, re-pin on merge. Block the regression before it lands.

Start with a working evaluation.

Two commands bring up the stack and run a real scorecard — no API key, no model, no signup.

Quickstart Star on GitHub