Benchmarks

Benchmark mode runs the whole fixture suite across a roster and scores each reviewer against ground-truth labels. It answers the question that matters when choosing a roster: which reviewers actually find the planted bugs without crying wolf?

Precision, recall, F1

A finding matches ground truth when the file matches and the line is within tolerance (|line diff| ≤ 3 by default). From the true positives (tp), false positives (fp), and false negatives (fn):

• Precision = tp / (tp + fp) — of what it flagged, how much was real (low = noisy).
• Recall = tp / (tp + fn) — of the real bugs, how many it caught (low = misses bugs).
• F1 = 2·P·R / (P + R) — the harmonic mean; the headline ranking axis.

Fixtures & scoring

The current suite has four diff-based fixtures. The clean-baseline is a false-positive control: it contains no real bug, so the correct output is zero findings.

A clean baseline with no findings scores P = R = F1 = 1.0 — but only for successful, parseable runs (a reviewer that crashes or returns garbage does not get a free perfect score).

Reference leaderboard

F1 score by reviewer, from the reference benchmark run. The chart re-themes with light / dark mode.

Footnote: This is a historical run (4 fixtures × 3 runs = 12 calls per reviewer, ~$0.42 total) and the names above are shown as-recorded, which is why they still carry version suffixes (glm-5.1, qwen-3.6-plus, kimi-k2.6…). Reviewer keys went version-free on 2026-09-22 — those rows map to glm, qwen, kimi and so on. These scores predate the 2026-09-22 model refresh, but the rows are not all stale in the same way:

• Repointed — genuinely stale. qwen-3.6-plus, glm-5.1, gemini-or, minimax-m2.7, mimo-v2-pro and kimi-k2.6 now run newer models (and mimo's old slug had been delisted outright), so these numbers measure something the reviewer no longer is.
• Renamed only — same model. grok-4.20 is now keyed grok-longctx and still runs x-ai/grok-4.20. The key moved; the model did not. hermes-4.3 is the same story with a wrinkle: its version lived in the key, while the route that actually ran was its OpenRouter fallback nousresearch/hermes-4-405b — unchanged today. Its direct primary was corrected (Hermes-4.3-36B, a slug never confirmed to exist), but that route needs NOUSRESEARCH_API_KEY, which this environment does not have, so it cannot have served the row.
• Unchanged. deepseek-v3.2 is still its own registry entry on the same slug — it was not replaced by deepseek, which is a separate reviewer.
• Unverifiable. opencode and codex route through a CLI subscription with no pinned slug, so nothing records what served them that day. Almost certainly stale, but this repo cannot demonstrate it.

Re-run --benchmark before acting on the ranking either way. With only four fixtures the leaderboard was noisy to begin with — treat it as directional.

Running a benchmark

# Smoke test one fixture first (catches config bugs in ~30s):
python scripts/benchmark.py --runs 1 --fixtures sql-injection --profile panel

# Full run across the suite:
python scripts/benchmark.py --runs 3 --profile panel
python scripts/benchmark.py --runs 3 --roster "glm,minimax,opencode"

Average call time is a cost / quality trade-off column, not a ranking axis: a fast, cheap frontier reviewer wins over a slow, expensive one when F1 is comparable.