The bet
Most AI code reviewers fail the same way: they're noisy, people mute them, and the one real injection bug goes by unread.
ReviewBot makes a stronger bet. Its required check, reviewbot/majors, stays red until a named developer records a decision on every major finding: ack it, or dismiss it with a reason. It doesn't clear on a fix. It clears on a decision.
Why attack it
Unit and integration tests passed, and the design was written down as a set of rules the system must never break. But tests check the cases you thought of. What I didn't have was evidence of how the deployed system behaved when developers did ordinary, untidy things: rebase, amend, delete a file, ack the wrong finding.
So I wrote down every scenario I could trigger and ran them all against the live service.
The method
- A dedicated test repo, 28 pull requests opened across three base branches, each driven to completion.
- Every scenario that could be triggered without a second GitHub account, a human at the UI, or a container restart.
- Each outcome checked in two places: what GitHub showed, and what Postgres recorded, read afterwards from inside the private network.
That second check mattered more than I expected.
What held
Branch targeting, the meaningful-change filter, debounce, the command surface and the report router all behaved as specified. Dismissals were final: a second dismiss, or an ack after a dismissal, was refused with "dismissals are final."
What broke
Nine defects. Three would block or mislead a developer in ordinary use.
Gated on the file, not the change
A later PR changed one unrelated line in a file with an old, undecided secret leak, and its author had to ack a leak they never wrote. The finding was correctly labelled "Not introduced by this change"; the gate simply didn't read it. That needed a ruling, not a patch.
A deleted file gated forever
A PR introduced a major, then a later push deleted the whole file. A deleted file is never re-analysed, so no push could clear it. The only way out was to ack a bug that no longer existed.
Every rebase, unmergeable
Amend, rebase or merge main in, and ReviewBot correctly skipped the redundant review but never published a check on the new commit. The day the check was required, every rebase would block its PR permanently.
Plus:
- An
ackposted on one PR cleared a finding belonging to another PR. - A review could post onto a PR that was closed mid-run.
- A file added and then moved produced duplicate findings.
- Re-acking wrote a second decision row.
- GitHub's "Re-run checks" paid for a full re-review.
Reading the tables changed the story
The database pass overturned two of the three "missed bug" reports in the first draft. Both defects had been found, but rated minor, which is why nothing appeared on the PR. The system was doing its job; GitHub alone made it look blind.
It also surfaced two defects that leave no trace on GitHub at all, such as a "gates" flag set to true on minor and suppressed findings that can never legally block anything. They had no effect yet, but they would quietly corrupt every metric built on that table.
Fix, redeploy, re-verify
All nine were fixed and deployed. The two that could be checked against the live service were re-run on it and passed. The report was kept as a record of what the run found, not as an open to-do list.
What it is now
Baseline, PR, weekly, monthly
A baseline at onboarding, a fast check on every PR, a weekly scan of what merged, and a monthly full-codebase pass.
Findings that survive a move
Each finding is fingerprinted on its enclosing function or class, not line numbers, so moved code keeps its history.
One container
It runs in one container for about $41/month at its planned volume.