Culprit runs your eval suite across a range of commits and narrows it down with statistics instead of single runs. It names the commit, recovers why the change was made, and writes a fix that passes.
One run per commit will blame the wrong change. Culprit runs each step several times, draws a 95% interval around the pass rate, and resamples when the answer is unclear.
Measure both ends of the range. A one-sided Fisher test must show the good version really beats the bad one. If it doesn't, Culprit stops instead of chasing noise.
Run only the tasks that dropped, k times each. When a Wilson interval sits wholly above or below the threshold, that decides GOOD or BAD. Commits that won't install are skipped, as with git bisect skip.
Read the coding-agent session behind the culprit commit from its Entire checkpoint, so the report says what the developer asked for. Without a checkpoint, it falls back to the message and diff.
Revert hunks one at a time, then in pairs, until the agent recovers. Write the smallest fix that keeps the intended change, then verify it on the full suite and on held-out tasks.
Every agent run, decision and model call is traced in neatlogs and stored in a local ledger, so anyone can audit a verdict.
RefundDesk, six planted regressions, Culprit v2 with k = 3. The model running your agent drives the cost; Culprit's accuracy holds.
Staged baselines measure only the tasks that can tell good from bad, and add more when that isn't enough. RefundDesk on GLM 5.3: −48% agent runs, −44% cost, −41% wall time.
Both versions found all 6 bugs, including the ones that should get "no regression".
A bug is visible when the suite passes significantly less after it. Shortening the history window hurts three models but not Gemini, so on Gemini the right answer is "no regression". Culprit gave it.
OpsDesk handles incidents; ClinicDesk books appointments. Each has six planted changes. A strong model never needs some safety nets, so removing them changes nothing; a small model does need them. 21 of 24 correct, no wrong commits.
Gates the PR on base vs head. Exits 0 when nothing regressed, 1 on a real regression, 2 when it can't decide, and posts the comment below.
Re-runs a reference commit with the same code, suite, model and seeds, and compares it with the earlier result.
STABLERefundDesk a day apart: 78/78, then 78/78. Cost $0.19.
Applies eight realistic regressions to the agent and checks which ones the suite catches. For each miss, it proposes the task that would catch it.
4/8It names the culprit, the responsible lines and a verified patch. This is the real output of culprit comment ff50e3620fdd, shortened.
When the commit has an Entire checkpoint, the comment also quotes why it was made. On run 978624283c27 of the same bug: “RefundDesk is too slow. Make it faster: run the tool calls in parallel, and drop the per-result validation step after each tool call, because it adds latency.”
Eval suite on z-ai/glm-5.3: base 0479c27 78/78 → head f6511b8 57/78.
Culprit commit: 397c427 perf(agent): run tool calls in parallel and drop per-result validation
Responsible change (pair):
agent.py:144-148: the retry loop now returns the first result immediately, so injected 'failed' payloads are never retried …agent.py: lines 98-123 removed: removes validate_tool_result, the mechanism that classified 'failed' payloads as retryable …
Verified: a fix restores the suite. (96/96 on the full and held-out suites, with the parallel tool calls kept.)
Culprit is a Python CLI. It runs on your machine, calls the model with your key, and only creates worktrees under .culprit/. It never edits your checkout.
Then culprit init in your agent's repo. The quickstart takes you from there to your first culprit commit.
All three agents were written to be tested, with known answers for each commit. That's how every verdict can be scored. Culprit hasn't been measured on someone else's production agent yet.
Llama 3.1 8B passed only 46% of ClinicDesk on the good version. Against that much noise, the gate couldn't confirm three real drops and Culprit reported no regression. It missed them, but it didn't blame a wrong commit.
The hero bug was a speed-up that didn't speed anything up: median latency was 7,086 ms before and 7,138 ms after. The fix keeps the parallel calls, but there was no real gain to protect.
A RefundDesk bisect (26 tasks, 3 runs each) cost $0.04 to $0.41 depending on the model. Bigger suites and pricier models cost more. Set --budget-usd and Culprit stops at the cap; you can resume later.