CI and pull requests

culprit ci is a regression gate for pull requests. It measures the PR's head against its base. If the head is significantly worse, it bisects the PR's commits, names the responsible hunk and writes a comment for the PR.

What it does#

1. Runs your suite at the base and the head, and applies the regression gate.
2. No significant drop: writes a short "no regression" comment and exits 0.
3. A real regression: bisects the PR's commits, localises the hunk (unless --no-localise), optionally writes and verifies a fix (--fix), writes the comment and exits 1.
4. Anything that stops the run (budget, an API outage, an error): writes what happened and exits 2. An outage never blames a commit.

Here is a real comment, from a run on our test agent, shortened:

### ❌ Culprit: this PR regresses the agent

Eval suite on `z-ai/glm-5.3`: **base `0479c27` 78/78** → **head `f6511b8` 57/78**.

**Culprit commit:** `397c42743b` perf(agent): run tool calls in parallel and drop per-result validation

**Responsible change** (pair):
- `src/refunddesk/agent.py:144-148`: This is the direct cause: the retry loop now returns the first
  result immediately, so injected 'failed' payloads are never retried, …
- `src/refunddesk/agent.py: lines 98-123 removed`: Removes validate_tool_result, …

**Verified:** a fix restores the suite.

<details><summary>Proposed patch</summary> … </details>

When the culprit commit has an Entire checkpoint, the comment also quotes why it was made:

**Why it was made** (Entire checkpoint `01M4JM8P2C4NDS2AAQ0MPVR8YR`):

> RefundDesk is too slow. Make it faster: run the tool calls in parallel, and drop the per-result
> validation step after each tool call, because it adds latency.

GitHub Actions#

Add your model key as a repository secret (OPENROUTER_API_KEY or ANTHROPIC_API_KEY, and optionally NEATLOGS_API_KEY), commit your culprit.toml, then add this workflow as .github/workflows/culprit.yml:

name: culprit

on:
  pull_request:

permissions:
  contents: read
  pull-requests: write

jobs:
  regression-gate:
    runs-on: ubuntu-latest
    timeout-minutes: 60
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0          # the bisect needs the PR's commits

      - uses: astral-sh/setup-uv@v6

      - name: Run the regression gate
        id: gate
        env:
          OPENROUTER_API_KEY: ${{ secrets.OPENROUTER_API_KEY }}
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
          NEATLOGS_API_KEY: ${{ secrets.NEATLOGS_API_KEY }}
        run: |
          set +e
          uvx culprit-cli@0.1.0 ci \
            --base ${{ github.event.pull_request.base.sha }} \
            --head ${{ github.event.pull_request.head.sha }} \
            --suite-ref ${{ github.event.pull_request.base.sha }} \
            --budget-usd 3 \
            --comment-file culprit-comment.md
          echo "exit=$?" >> "$GITHUB_OUTPUT"

      - name: Post the result on the pull request
        if: always()
        env:
          GH_TOKEN: ${{ github.token }}
        run: gh pr comment ${{ github.event.pull_request.number }} --body-file culprit-comment.md

      - name: Fail the check on a regression
        if: steps.gate.outputs.exit != '0'
        run: exit ${{ steps.gate.outputs.exit }}

Why these choices#

• fetch-depth: 0: by default Actions checks out a single commit, and there would be nothing to bisect.
• --suite-ref set to the base: the PR is judged by the suite it started from, so a PR can't make itself pass by editing the tests.
• culprit-cli@0.1.0: pinning the version keeps CI results comparable over time. Upgrade on purpose.
• Failing on exit 2 as well: the last step fails the check on 1 (regression) and on 2 (couldn't decide). To fail only on a confirmed regression, change the condition to steps.gate.outputs.exit == '1'.

Cost per pull request#

A clean PR costs one measurement of the base and the head. With staged baselines, that's one run of every task on each end, plus k runs of the tasks that differ. A regressed PR also pays for the bisect and localisation. --budget-usd caps the whole job; set it from what a bisect costs on your model.

Other CI systems#

culprit ci is a plain command with exit codes, so it works in any CI that can install uv and run shell commands. Give it a full clone and your model key in the environment, run the same command, and post culprit-comment.md wherever your reviews happen. To rebuild the comment for any finished run, use culprit comment.

Options#

Exit code 0 when clean, 1 on a regression, 2 when the run stopped.