Your agent passed its evals last week. Today it fails some of them, and dozens of commits sit in
between. Culprit is git bisect for that situation, built for noisy evals: it repeats every
measurement, decides with confidence intervals, and blames nothing when the drop could be noise.
It runs on your machine with your own model key, needs no SDK in your agent, and records every decision so you can check it.
The workflow#
| Step | Command | What you get |
|---|---|---|
| 1 | culprit init |
Your key saved, your repo set up. |
| 2 | culprit run |
The commit that made the agent worse, and why it was made. |
| 3 | culprit localise |
The lines in that commit that did it. |
| 4 | culprit fix |
A minimal fix, verified on your suite and held-out tasks. |
| 5 | culprit report |
The receipts: every step, interval and cost. |
Beyond one-off bisects: culprit ci gates every pull request,
culprit drift tells you whether the model behind the API changed, and
culprit audit shows which regressions your evals would miss.
Results so far#
On three test agents with planted regressions and known answers, Culprit ran 54 scored bisects across 5 victim models: 51 correct, 3 missed, none naming a wrong commit. Limits explains the misses and what these results don't show.
Start here#
- Quickstart: install and run your first bisect.
- Prepare your agent: the eval module and suite Culprit expects.