Auditing your evals · Culprit docs

Mutation testing for agents. Culprit makes realistic regressions and shows which ones your eval suite would miss.

A bisect can only find a regression your evals can see. culprit audit tells you which realistic regressions they can't see, before one ships.

Terminal

culprit audit --ref HEAD --out audit.md

How it works#

For each mutation operator, a model reads your agent's source and writes a small, plausible change of that kind as exact edits, with a one-line summary. If the pattern doesn't apply to your agent, it says so.
Each change (a mutant) is applied in its own throwaway worktree. Nothing is committed anywhere.
Your suite runs on each mutant. A mutant is caught when the suite passes significantly less than on the unchanged ref (one-sided Fisher test, p < 0.05), and missed otherwise.
The mutation score is caught ÷ valid mutants. A mutant that doesn't apply or no longer installs is reported invalid and left out of the score.
For every missed mutant, the report suggests an eval task that would catch it.

The operators#

Each is a regression a developer could plausibly make, with the reason they'd give:

Operator	The change	The plausible reason
`drop_result_validation`	Remove the check on a tool's result before the agent acts on it.	It adds latency.
`disable_retries`	Stop retrying failed or timed-out tool calls.	Retries add latency.
`shrink_history`	Show the model only the last 2–4 messages.	Token cost.
`drop_prompt_rule`	Delete the most important business or safety rule from the system prompt.	The prompt is too long.
`remove_code_guard`	Remove a code check that blocks a disallowed action.	The prompt already says it.
`lenient_tool_args`	Fill missing tool arguments with defaults instead of rejecting the call.	Fewer rejected calls.
`half_renamed_param`	Rename a tool parameter in its schema only, as an unfinished refactor would.	Naming consistency.
`vague_tool_description`	Rewrite a tool's description so it no longer says when to use it.	Simplify the tool list.

culprit audit --list prints them. Choose some with --operators drop_prompt_rule,remove_code_guard.

A real result#

On our refund agent, the suite caught 4 of 8:

Mutant	Verdict	Passes
Drop tool-result validation	caught	61/78
Disable retries	caught	53/78
Shrink the conversation history	caught	72/78
Half-done parameter rename	caught	15/78
Drop a business rule from the prompt	missed	78/78
Remove a code-level guard	missed	78/78
Accept malformed tool calls	missed	78/78
Vague tool description	missed	78/78

For "Remove a code-level guard", it suggested a task where a customer with a $1,250 order demands a full refund and refuses escalation, which passes only if no refund over $500 is issued. That's exactly the test the suite was missing.

Your own mutants#

Terminal

culprit audit --patches my-mutants/

Every *.patch file in the directory is a mutant. No model is called, so the result is deterministic.

Cost and resuming#

An audit runs your suite once per mutant plus once on the unchanged ref, at k repetitions each, plus one model call per operator. Ours cost $1.80 for 8 mutants. --budget-usd caps it.

The proposed mutants are stored before anything runs, so culprit audit --run-id <id> resumes a stopped audit without asking the model again.

Options#

Option	Default	Description
`--ref REF`	`HEAD`	The version of the agent to mutate.
`--operators LIST`	all	Comma-separated operator names.
`--patches DIR`	none	Use your own `*.patch` mutants instead. No model calls.
`--run-id RUN_ID`	none	Resume this audit; stored mutants are reused.
`--out PATH`	none	Write the markdown report here.
`--list`	off	List the operators and exit.
`--k N`	3	Repetitions per task.
`--budget-usd X`	3.00	Hard cost cap.
`--repo`, `--eval-module`, `--suite`, `--suite-ref`	as for `run`
`--workers N`	4	Agent runs in parallel.

Exit code 0 when the audit completes, 2 otherwise.