Do your review bot's warnings turn into bugs? What your history can and cannot tell you

Most teams I talk to switched on an AI reviewer some time in the last year. It comments on every pull request within minutes, the comments read well, and some of them catch real problems. When the renewal comes round, the conversation runs on feel: it caught that one thing, or it is noisy.

The question I would want answered before renewing is narrower. When the reviewer flags something and the author moves on, was the flag right? If ignored warnings keep turning into bug fixes a month later, the bot is useful and the team is not listening. If they do not, the comments are noise however many threads get marked resolved. And the same question, asked of the human reviewers in the same repository, is the only fair yardstick, because every team already has an opinion of its reviewers.

This post is about how to ask that of your own history: four numbers, where each comes from, and what it takes for the hardest of them to mean anything. I tried the method on three open-source projects to see whether it holds up, and to get a rough gauge of where a typical setup sits. The projects are not the point, and neither are the review tools; the method is.

The four numbers

Heeded. Of the lines a comment points at, how often did they change before merge? This is what most published studies measure, in one form or another, and it is only “did anyone react”.

Heeded beyond chance. Code under review keeps moving: authors rework, rebase and answer other comments. Some flagged lines would have changed anyway. So for every comment, take a placebo: a stretch of lines the same pull request added, of the same length and on the same commit, that nobody commented on. Only the difference between the comment and its placebo counts. Inverted, it is the number of comments a person reads for each change a comment causes, which is the figure to set against what the bot costs.

Right when ignored. For flags nobody acted on, how often were those lines fixed again within 90 days of merge by a commit that fixed a bug, beyond how often the placebo lines were? This is the number that answers the question above. It needs two things most repositories do not have by default: a reliable way to tell a bug fix from any other change, and enough ignored flags for a small difference to show.

The resolve button, for comparison. How often threads are marked resolved, next to the first number. The gap is how much the button flatters.

Each number is computed for the AI reviewer and for the human reviewers of the same repository, side by side, and split by front end against back end and by bug fixes against new capability where the repository’s labels allow.

What I wrote down before running it

Four predictions, so that the method could fail in public: human comments are heeded at least twice as often as the bot’s; a bot warning nobody acted on predicts a later bug fix no better than chance; a human warning nobody acted on does predict one; and the bot does better on back-end bug fixes than on new front-end features. Three of the four were wrong, and the way they were wrong is the useful part.

How I tested the method

Three public repositories. Sentry, Home Assistant and a large observability project whose trademark policy does not allow me to name it. Each runs one or more AI reviewers next to active human review: a third-party review app, GitHub’s built-in reviewer, the project’s own. I took the newest 1,500 merged pull requests from the first half of 2026 that carried a review comment, about 21,000 comments in all, six in ten of them from bots. No model calls and no money; a weekend of fetching, mostly waiting out GitHub’s hourly limits.

Checks before reading anything. A fixture repository with known history that the tool had to score exactly; a null run in which a second placebo stands in for every comment, so every difference must come out at zero; and a recount of comments through a different route into GitHub. The null run caught two biases in my own method before I had seen a single real result, which is what it is for.

A bug-fix signal per repository. The tool reads “this commit fixed a bug” from the labels or titles a team uses, under one rule: bug labels if at least one in ten later pull requests carries one, otherwise titles that start with “fix” at the same bar, otherwise no signal. Which one a repository gets is itself a finding.

What the three repositories showed

The baseline is the placebo row: what a comment on random lines of the same pull request would score.

People are heeded more, but not twice as much. Between 40% and 57% of AI comments were followed by a change to their lines before merge, against 60% to 79% for human comments. People lead in every repository, by a factor of 1.1 to 1.5 rather than the two I predicted.

A third or more of that would have happened anyway. Take the placebo away and a person reads about three to four AI comments for each change one causes, against two to two and a half for a human comment. The spread between the three setups is as large as the spread between bots and people, and the noisiest setup produced six bot comments per reviewed pull request, ten on pull requests that added new capability.

The resolve button flatters the bot by 20 to 45 points. On one repository more than half of the resolved AI threads were resolved by a bot, tidying up after itself.

Right when ignored: the method works and the history does not. On the two repositories with a usable bug signal, lines a bot flagged and nobody changed were fixed later no more often than their placebo, an excess of about one point with an interval through zero. The same was true for people, which I did not predict. Reading 30 matches per repository by hand, the later fix answered the comment in 4 of 30 on each. The good cases are very good: a bot warned of a race when two webhooks arrive out of order, and a later pull request was titled “Stub PullRequest on check-event race”. Most matches are unrelated work on the same lines, and a “bug fix” can be a missing word in a sentence. Open-source history has nothing that says “this reached production and hurt someone”, so that is as sharp as its bug signal gets.

And the volume was not there either. Flagged lines get a later fix 1% to 8% of the time. To show a one-point difference above that, you need roughly 5,000 to 12,000 ignored flags per reviewer kind; a three-point difference needs 400 to 1,500. These repositories had 500 to 2,600 per kind, enough to have seen a difference of about one and a half to five points, which did not appear. So either the effect is smaller than that or the bug signal is too blunt to show it, and public history cannot say which. The splits by kind of change fared the same: every difference sat inside the noise.

Against the industry, as a gauge

On the closest comparable measure, one vendor’s comments inside one company changed the exact line 29% to 43% of the time; the 40% to 57% here is at or above that, and a year newer. The study that compared with humans found about 60% for people and 1% to 19% for the AI review actions it studied a year or so earlier, with a stricter, model-judged measure. The resolve-button figures published for coding agents, around 70%, sit right where the resolve button sits here, which is the point: that measure is the flattering one. None of the published figures corrects for churn, so none can be compared with the second row. If your repository prints something near these numbers, your setup is roughly where the field is; the second row is where you can pull ahead by tuning the bot’s volume.

What I’d tell a fellow technical leader

Pick the four numbers, and insist on the placebo. Heeded, heeded beyond chance, right when ignored, and the resolve rate for contrast. Without the placebo the first number flatters everyone by a third.

Price the bot in comments read per caused change. Three to four against two to two and a half is a cost paid in your reviewers’ attention, and it moves with the bot’s settings more than with the bot.

Treat the resolve button as a vanity metric. It runs 20 to 45 points ahead of the code and can be pressed by bots.

Fix your bug signal before asking the hard question. The third number is only as sharp as the way your team marks bugs. Link fix pull requests to incidents or customer-reported bugs, and check that at least one in ten later pull requests carries that mark. Open source cannot do this; you can.

Let the tool tell you how long to wait. From your own base rate it prints how many ignored flags you need to see a one, two or three point difference, and the smallest difference your current history could have shown. Collect until the number you care about is in reach, then read 30 matches by hand before believing it.

Try it yourself

The kit now includes review_bot.py. Write a short config naming your AI reviewer accounts and your bug labels, give it a read-only GitHub token and a clone of your default branch, and it prints the four numbers for your repository, by reviewer and by split, with the sample-size rule and 30 comments to read by hand. It needs only standard Python and git and makes no model calls. On a repository the size of these, expect a few hours of fetching.

What’s next

The next post goes back to the start of delivery: which tickets you can hand to an agent at all, judged from the ticket alone, on public tickets that agents have already attempted.

Appendix: method, data and limits

The study was written down on 14 October 2026, before any data was fetched, and revised once that day when the first draft read as a test of how studies count rather than a method a team can run. Every change after that was recorded with its reason before the affected result was read. The main ones: GitHub’s GraphQL interface returns bot names without their “[bot]” suffix, which the tool now restores; the null check showed that a placebo must be untouched before merge, like the ignored comment it is compared with, or the comparison is biased; and because Sentry hardly uses bug labels, the bug signal is chosen per repository by the one-in-ten rule (labels, then “fix” titles, then none). The churn-corrected heeded rate, comments read per caused change, the share of threads resolved by bots and the sample-size rule were added after the first results were in and are reported as such; they are now part of the tool.

Comments: inline only, first in their thread, on the new side of the diff, not by the pull request’s author. AI reviewers per repository were listed from account names before any outcome was computed; linters, dependency, CI and security bots are excluded. Several AI reviewers in one repository are counted together. The newest 1,500 pull requests per repository gave windows of late April to June 2026 for Sentry and Home Assistant and February to June for the third. Comments on commits that no longer exist on GitHub were left out: 730 on Sentry, 167 on the observability project. 95% intervals come from 2,000 bootstrap draws over whole pull requests; a cell with fewer than 100 ignored AI flags is reported as too thin. The sample-size rule is the usual two-proportion calculation at 95% confidence and 80% power.

Limits: a line change is not agreement; a later fix is not proof that the flag was about that defect, as the hand reading shows; the bug signal depends on each team’s habits and differs between Sentry (titles) and Home Assistant (labels); 90 days misses slower defects; people choose which lines to comment on and bots comment widely, so this compares reviewers as teams use them, not their judgement on equal terms. The hand reading was done by Claude at my request, with the same procedure for all 60 comments; I checked 12 of its verdicts and agreed with all of them.

Data: public pull requests, review comments and commits from three repositories, fetched through the GitHub API and reported in aggregate only, with no authors or reviewers named. Review tools are described by category rather than by product.

I am not affiliated with Sentry, Home Assistant or the third project, or with any of the review tools, and none has reviewed or endorsed this post.

This is personal research, done in my own time and on public data only. The views are my own and not my employer’s.

I used Claude to help build the tool and draft this post.