How bisect works · Culprit docs

How Culprit decides that a regression is real, picks the tasks to run, and calls each commit GOOD or BAD despite noisy evals.

git bisect assumes each commit is simply good or bad. Agent evals aren't like that: the same commit can pass a task on one run and fail it on the next. A bisect that trusts single runs will blame the wrong commit. Culprit treats every step as a measurement with uncertainty.

The run, step by step#

Pin the suite. Culprit snapshots the suite at suite_ref once and uses it for every commit, so old commits are judged by the same tests as the new ones.
Measure both ends. The good and bad refs each run the suite. Culprit first screens every task once on both ends, then runs the remaining repetitions (k, 3 by default) only for tasks that didn't pass on both ends. Those are the tasks that can tell good from bad.
Gate. A one-sided Fisher exact test asks whether good really passes more than bad. If not (p ≥ 0.05), the run ends as no_regression and nothing is blamed.
Choose the telling tasks. Tasks are ranked by how much their pass rate fell from good to bad, and the top 8 that fell are used for the bisect. Tasks that pass on both ends would only add cost.
Bisect. Culprit tests commits between the ends, halving the range each time, and decides each one with the rule below. A commit that won't install is skipped.
Explain. For the first bad commit, Culprit records why it was made. See culprit run.

The decision rule#

At each commit, Culprit runs the telling tasks k times and counts passes. It computes a 95% Wilson score interval for the pass rate, and compares it with a threshold: the midpoint between the good and bad pass rates over the same tasks.

The whole interval is at or above the threshold: GOOD.
The whole interval is at or below it: BAD.
The interval straddles it: inconclusive, so Culprit runs more.

Output

  464e05fa11 'revert requests pin': 14/15 CI [0.70, 0.99] vs t=0.50 → GOOD
  462db36ee1 'docs: maintenance note 11': 1/15 CI [0.01, 0.30] vs t=0.50 → BAD

An inconclusive step is resampled up to two more rounds. The first round adds repetitions of the same tasks; the second adds other tasks that also dropped, or more repetitions if there are none. If it's still unclear after that, Culprit takes the side the pass rate falls on and flags the step low_confidence in the report.

Why a Fisher test for the gate#

Checking whether two 95% intervals overlap is much stricter than a 5% test. 57/60 against 48/60 overlaps, so a regression that breaks 3 of 20 tasks would be called noise. The one-sided Fisher exact test on the same counts is the right question: how likely is a gap this large if both ends had the same pass rate?

Skipped commits#

Before testing a commit, Culprit creates a worktree, runs uv sync --locked and checks that the eval module imports. If either fails, the commit is SKIPped and its neighbour is tested, as git bisect skip does. A run where skipped commits sit right at the boundary ends as ambiguous, with the list of commits that could be the culprit.

How a run ends#

The possible results, from found to infra_error, are listed on the culprit run page.

Resuming and reusing#

Every agent run, step and decision is written to the ledger as it happens, so a stopped run can continue without measuring anything twice. See culprit resume.

Later runs also reuse baselines: if an earlier run already measured the same commit with the same suite, model and k, those results are copied in rather than run again. The --reuse, --staged, --k and --workers options are described on the culprit run page.