What Culprit needs#
- A Python agent in a uv project with a committed
uv.lock. Agents built with pip, Poetry or other languages aren't supported yet. - An eval suite that sees the problem. Culprit can only find a regression your evals detect.
culprit auditshows what they miss. - A good ref. Culprit compares against a version you know was better.
What it can't tell apart#
- A model change from a code change, in one run. Use
culprit driftfirst when nothing obvious changed. - A weak model's noise from a real drop. If the agent fails often even at the good ref, the gate
may not be able to confirm a drop, and Culprit reports
no_regressionrather than guess.
Safety#
- Culprit runs the repo's code at every commit it tests, with your keys in the environment, as running its test suite would. Only run it on repositories you trust.
- Proposed fixes are never committed. Review the patch before applying it.
What our measurements don't show#
We measured Culprit on three agents we wrote for testing (refund support, incident response and appointment booking), with planted regressions and known answers. The Overview has the totals.
- The bugs were planted. That's what made every verdict checkable. Culprit hasn't yet been measured on someone else's production agent.
- The 3 misses came from Llama 3.1 8B on the booking agent, which passed only 46% of tasks even at the good version. Against that noise the gate couldn't confirm three real drops.
- One "faster" commit wasn't faster. The hero bug was a speed-up that didn't measurably speed anything up. The fix kept the parallel calls, but there was no real gain to protect.
- Your cost will differ. Costs scale with suite size,
k, the model and the range.