culprit: command not found#
uv installed Culprit, but its tool directory isn't on your PATH. Run uv tool update-shell and open
a new terminal, or use uvx culprit-cli … instead.
"no eval module … run culprit init there, or pass --eval-module"#
Culprit didn't find a culprit.toml in this directory or its parents, up to the git root. Run
culprit init at your repo's root, or pass --eval-module. If init itself can't detect the
module, it looks for files named eval.py, evals.py or eval/__main__.py outside tests/ and
.venv/. Pass --eval-module your.module if yours is named differently.
"no model key"#
No ANTHROPIC_API_KEY or OPENROUTER_API_KEY in your shell, your repo's .env, or the saved setup.
Run culprit init without --yes to save one. In CI, add the key as a secret and pass it in the
job's environment.
"… is not a … model id"#
VICTIM_MODEL or CULPRIT_MODEL names a model from a different provider than your key selects.
OpenRouter IDs look like vendor/model; Anthropic IDs have no slash. If both keys are set, Anthropic
wins: unset ANTHROPIC_API_KEY in that shell to use an OpenRouter model.
Every commit is skipped#
Each commit is installed with uv sync --locked, then python -c "import <eval_module>" is run.
Common causes:
- No
uv.lock, or one that's out of date withpyproject.tomlat that commit. Runuv lock, commit it, and use a range after that commit. - The eval module does work on import, such as parsing arguments, and fails. See Prepare your agent.
- A private dependency that the machine can't fetch.
The last part of each skipped commit's install output is stored with its step in the ledger, and in the neatlogs trace when tracing is on.
"eval runner printed no JSON object"#
Your module's last JSON line couldn't be found in its output. Run it by hand from the repo root, for
example uv run python -m myagent.eval --task <id> --seed 0, and check that the last JSON object it
prints is on a single line. Culprit treats this as a transient error and retries twice; if it keeps
happening, the run stops with infra_error and nothing is counted against a commit.
infra_error: rate limits, timeouts or outages#
Your provider kept failing after retries, or reported a fatal error such as an invalid key, no credit
or an unknown model. Fix the cause, then culprit resume. Lowering --workers makes
rate limits less likely.
no_regression, but the agent really is worse#
The gate needs evidence that good passes more than bad. That can fail when:
- The suite doesn't see the problem. Run
culprit auditto find blind spots. - The model is too noisy. If the good ref doesn't pass reliably either, a drop is hard to confirm.
Raise
--k, or use a stronger victim model. - Nothing in your code changed. Check for drift.
budget_exceeded#
The run reached its cost cap. Continue it with culprit resume.
"ledger total reached the $75.00 global cap"#
See the ledger-wide cap.
"culprit: using … from …/.env"#
Not an error. A setting came from the repo's .env rather than your shell or saved setup. See
Which source wins.
Results differ between runs#
Agent evals are noisy, which is why Culprit repeats tasks and uses intervals. Two runs can take
different paths and still agree on the culprit. Seed your agent's randomness with --seed, and report
passed from deterministic checks where you can.