Drift checks · Culprit docs

Find out whether your agent got worse because of your code, or because the model behind the API changed.

Sometimes nothing in your repo changed and the evals still got worse. Providers update models behind the same name, and tools and services change. Bisecting your history can't find a cause that isn't in it. culprit drift answers that question first.

Terminal

culprit drift --ref v1.0

How it works#

An agent run is determined by the commit, the suite, the model, the eval module and the seed. Drift re-runs a ref that Culprit has already measured, with exactly the same tasks, repetitions (seeds), suite snapshot, model and k as that earlier measurement. Only time differs. Then it compares the two with one-sided Fisher tests in both directions:

Result	Meaning	Exit code
`STABLE`	No significant change. If your agent is worse now, look in your history: run `culprit run`.	0
`IMPROVED`	The same code passes more than it did.	0
`DRIFTED`	The same code passes less than it did. The cause is outside your repo; bisecting won't help.	1

It also lists which tasks changed. A real check on our test agent, a day apart:

Output

STABLE: no significant change. 0479c273fb: then 78/78 (run ff50e3620fdd), now 78/78;
one-sided Fisher p(worse) = 1.0000, p(better) = 1.0000

That check cost $0.19.

What it needs#

An earlier measurement of the ref. Every culprit run records the baseline of its --good ref, so run a bisect first. Drift picks the oldest matching measurement; --against <run_id> chooses another.
The same model and k. Drift only compares like with like. If you've changed VICTIM_MODEL or --k since, there's nothing to compare with, and it says so.
The ref must still install. If it doesn't, that's a change in your environment, not drift, and the check stops.

Nothing from the earlier measurement is reused: the point is to measure again.

When to run it#

Before a bisect, when your agent got worse but you can't see why.
On a schedule, for example nightly in CI against your last release, to notice provider changes early.

Terminal

culprit drift --ref v1.0 --budget-usd 2

Options#

Option	Default	Description
`--ref REF`	required	A known-good ref with a recorded baseline.
`--against RUN_ID`	the oldest	Compare with this run's baseline of the ref.
`--k N`	3	Must match the recorded baseline.
`--budget-usd X`	2.00	Hard cost cap.
`--repo`, `--eval-module`	as for `run`
`--workers N`	4	Agent runs in parallel.

Exit code 0 for stable or improved, 1 for drifted, 2 otherwise.