AI code review for pull requests: what a review bot has to get right

Most AI review bots get muted within a week. Not because the findings are wrong, but because they arrive as a wall of text, repeat themselves on every push, and never say whether the pull request is good to go. Here is what we think a reviewer has to do to stay on the team, and how quellbot does each one.

1. Say whether it should merge

A review that ends without a verdict pushes the decision back to the person, which is the work the review was supposed to save. quellbot leads with one: Approve, Request changes, or Comment. A blocker submits Request changes. A clean diff submits Approve. The verdict is a real GitHub review state, so it shows up in the merge box like any teammate's.

Some teams do not want a bot's Approve to count toward branch protection, and that is a reasonable position. Set review.verdicts: false and every review lands as a plain comment; the findings are unchanged, only the state is.

2. Blockers first, nits last

A busy pull request can carry twenty observations. Listing them in file order buries the one that matters under nineteen that do not. quellbot opens with a summary, then a per-file walkthrough table, then findings grouped by severity: blocker, warning, convention, nit. A reviewer with two minutes reads the first group and knows whether to keep reading.

How much of the lower groups you want is a profile choice: chill drops nits altogether, default keeps them, and assertive also mentions minor wins it would otherwise leave alone.

3. Make the fix a click

When a finding has a concrete edit ("add the null check before line 88", "this should be const"), quellbot attaches it as a GitHub suggestion block. The author applies it from the diff view with one click and a commit is created in their name. The reviewer's job is to spot the problem; the fix should not be a second job.

4. Review the delta, not the whole PR again

This is where most bots lose the room. The author pushes a fix, and the bot re-reviews from scratch: the same twelve findings, plus one new one, and now nobody can tell which is which. quellbot re-reviews the delta. On a new push it posts what is fixed, what is still open, and what is new, resolves the threads you addressed, and never repeats a finding it has already made. The conversation stays the length of the actual disagreement.

5. Know the house rules

Correctness findings are only half of a review; the other half is "we do not do it that way here". quellbot reads the repository's own CLAUDE.md conventions before it reads the diff, and review.instructions in .github/quellbot.yml adds anything that file does not say. If the codebase reads the clock through clock.now() so tests can fake it, a raw new Date() gets a convention finding, with the reason.

review.ignore keeps generated files, lockfiles, and vendored code out of the review, so the findings are about code a person wrote.

6. Stay quiet when it should

Draft pull requests are not reviewed until they are marked ready. Very large diffs are skipped above a size threshold you set on the Triggers page, because a 4,000-line review helps nobody. And the review is on demand as well as automatic: comment /review to run it again, /explain for a cold-reader walkthrough of the change, or mention @quellbot on any thread to ask a question and get an answer that cites the file and line.

Where the review runs

Every review happens in an ephemeral machine in your own Fly.io account, on your own Claude or Codex credential. The diff is read there, the review is posted, and the machine is destroyed. quellbot's servers never hold a copy of your code. The security model explains why that is the design rather than a feature.

The configuration, in full

version: 1
review:
  profile: default          # default | chill (no nits) | assertive (also minor wins)
  verdicts: true            # blocker -> Request changes, clean -> Approve
  ignore:
    - pnpm-lock.yaml
    - "**/*.generated.ts"
  instructions: |
    We read the clock through clock.now() so tests can fake it.
    Errors cross module boundaries as Result values, never thrown.

Everything else, the lane, the model, the reasoning effort, is set in the console per account, with per-repository overrides.