culprit fix 77f60c9fccf8Run it after culprit localise on the same run ID.
What it does#
- Reads the evidence: the responsible hunks, why the commit was made, the failing tasks and the affected files.
- Names what the commit was for, such as lower latency or fewer tokens, so the fix can keep it.
- Proposes exact edits that restore the lost behaviour without reverting the commit. It may only touch the responsible hunks' files, never your suite.
- Verifies the fix at the bad head on the full suite plus your held-out tasks,
ktimes each. - Tries again with the failure as feedback if a proposal doesn't pass (2 attempts by default), then falls back to verifying a revert of the responsible hunks.
A fix counts as verified only if it passes three gates:
| Gate | Passes when |
|---|---|
| Quality | Its pass-rate interval reaches the good baseline's. |
| Intended gain kept | The named metric is no worse than at the bad head by more than 10%. |
| No new failures | No task that always passed at good always fails with the fix. |
The result#
| Status | Meaning |
|---|---|
verified_fix |
A proposed fix passed all three gates. |
verified_revert |
No fix passed, but reverting the responsible hunks did. |
Nothing is committed for you. See the patch with culprit comment, review
it, and apply it yourself.
Add held-out tasks
Tasks in evals/heldout/ are never used by the bisect, only to check that a fix generalises. A few
of them make a verified fix much more trustworthy.
Example
On our test agent, a "make it faster" commit had removed tool-result validation and broken the
retry loop. localise found those two hunks together, neither alone being enough. The fix restored
the validation, kept the parallel tool calls, and passed 96 of 96 runs on the full and held-out suites.
Options#
| Option | Default | Description |
|---|---|---|
--attempts N |
2 | Fix proposals verified before falling back to reverting the hunks. |
--tolerance X |
0.10 | How much worse than the bad head the intended-gain metric may get. |
--budget-usd X |
2.00 | Cost cap, on top of the run's cost so far. |
--workers N |
4 | Agent runs in parallel. |
--replay |
off | Use only what the ledger holds. |
Exit code 0 for verified_fix or verified_revert, 1 otherwise.
Next#
Read the report with culprit report <run_id>.