Prepare your agent

Culprit doesn't import your agent or need an SDK. At each commit it checks out a worktree, installs it, and runs your eval module once per task and repetition. Your repo needs three things.

1. An installable history#

• A git repository with a ref that was good (a tag, branch or commit) and one that is bad.
• A uv project with a committed uv.lock. Culprit installs each commit with uv sync --locked in its own worktree under .culprit/worktrees/. --locked is deliberate: a commit whose lockfile doesn't match its pyproject.toml is treated as broken, not quietly re-resolved.

A commit that fails to install, or whose eval module can't be imported afterwards, is skipped, the way git bisect skip works, and a neighbouring commit is tested instead. If every commit is skipped, see Troubleshooting.

2. An eval suite#

A directory with one file per task, evals/ by default. The file name without its extension is the task ID, so evals/refund_small_order.json is the task refund_small_order.

• Files ending in .json, .yaml, .yml, .py or .toml count. Names starting with _ or . are ignored.
• What's inside each file is up to you: Culprit only lists the names and passes the ID to your module.
• An optional heldout/ subdirectory holds tasks the bisect never uses. culprit fix runs them to check that a fix generalises.

evals/
  refund_small_order.json
  refund_timeout_once.json
  lookup_failed_no_refund.json
  heldout/
    refund_partial_order.json

The suite is pinned. Every commit in a run is judged by the same copy of the suite, taken from suite_ref (by default the branch you ran culprit init on). Culprit copies it into each worktree, replacing whatever that commit had, so an old commit is judged by today's tests.

3. An eval module#

A Python module that runs one task once and prints one JSON line. Culprit calls it from the root of each commit's worktree:

python -m myagent.eval --task refund_small_order --seed 0

and reads the last line of standard output that is a JSON object:

{"passed": true, "cost": 0.0026, "latency_ms": 7086, "tokens": {"in": 1840, "out": 212}}

The exit code doesn't matter; the JSON line does. Anything else your module prints is ignored.

What Culprit passes to your module#

Your module also gets your shell environment, your saved keys, and your repo's whole .env. Culprit removes VIRTUAL_ENV, UV_PROJECT_ENVIRONMENT and PYTHONPATH, so your agent runs in its own environment rather than Culprit's.

Report errors honestly#

A failed API call says nothing about your code. Culprit recognises these errors in the error field (or in the output when no JSON was printed) and never counts them against a commit:

• Transient (rate limits, timeouts, 5xx, overloaded): the run is retried twice, with a longer wait each time.
• Fatal (bad key, no credit, unknown model): the run stops with status infra_error. Fix the cause, then culprit resume <run_id> continues where it stopped.

So print the provider's error text, for example {"passed": false, "error": "RateLimitError: …"}, rather than turning an outage into a failed task.

A minimal example#

# myagent/eval.py
import argparse, json, os, pathlib, time


def load_task(task_id: str) -> dict:
    for folder in ("evals", "evals/heldout"):  # held-out tasks live in a subdirectory
        path = pathlib.Path(folder, f"{task_id}.json")
        if path.exists():
            return json.loads(path.read_text())
    raise SystemExit(f"unknown task {task_id}")


def main() -> None:
    from myagent.agent import Agent
    from myagent.checks import grade

    parser = argparse.ArgumentParser()
    parser.add_argument("--task", required=True)
    parser.add_argument("--seed", type=int, default=0)
    args = parser.parse_args()

    task = load_task(args.task)
    agent = Agent(model=os.environ.get("VICTIM_MODEL", "z-ai/glm-5.3"), seed=args.seed)
    start = time.monotonic()
    try:
        result = agent.run(task["message"])
        print(json.dumps({"passed": grade(task, result), "cost": result.cost_usd,
                          "latency_ms": (time.monotonic() - start) * 1000}))
    except Exception as exc:  # an API failure is not a failed task
        print(json.dumps({"passed": False, "error": f"{type(exc).__name__}: {exc}"}))


if __name__ == "__main__":
    main()

Check your setup#

culprit init --check

init detects your eval module, counts your tasks, writes culprit.toml, and runs one task exactly as a bisect step would. It reports the parsed result, or what went wrong.