Teaching NeetoBugWatch to disagree
The AI code reviewer I run approved most pull requests without a word. Rebuilding it as a panel of arguing reviewers fixed that, and the fix was mostly about where I spent the caution.
NeetoBugWatch kept telling everyone at Neeto their code was fine. By June I had stopped believing it, which is an awkward place to land when you are the one who built it. So I took two pull requests it had already reviewed in production, re-reviewed them with a small panel of AI reviewers, each handed a single angle to care about and told to miss nothing, and then checked everything the panel raised against the code. NeetoBugWatch had caught one real problem across those two pull requests, and had even approved one of the two outright, with nothing to say. The panel found twelve.
One out of twelve is not a rounding error. I had spent months making NeetoBugWatch smarter, and what I had built was cheap, fast, and mostly silent. It took me a while to work out why, and my first few fixes made it worse before anything got better. You don't need to know anything about AI or code review tools to follow along. I'll explain as I go.
What NeetoBugWatch is
NeetoBugWatch is an AI code reviewer. It reads a pull request the way a colleague would: goes through the changes, leaves comments on specific lines, and ends with a verdict, either "looks good" or "needs attention". It reviews every pull request across the Neeto product suite, roughly 2,300 a month across about 200 repositories, all on Google's Gemini models. The entire bill came to about 125 dollars a month; a typical review cost single-digit cents. I am the only engineer on it, which for this story means there is nobody else to blame.
Cheap, and quiet. Over a recent 45 day stretch of production data, 61 percent of its reviews produced zero findings. More than half of all pull requests got a clean bill of health. The average review produced 0.76 comments. For a tool whose entire job is finding problems, its most common output was nothing.
The obvious explanation would be that everyone at Neeto writes flawless code. The re-review above, plus every bug that reached customers, said otherwise.
Where the findings went
My first guess was that the findings were being generated and then thrown away by some overzealous filter. So I traced individual missed bugs through the pipeline, stage by stage, expecting to find the filter that ate them. For most of the misses there was nothing to trace. The reviewer had never produced the finding at any stage.
Over months of tuning, I had wrapped the reviewer in rules, each one added for a good reason, usually right after an embarrassing comment. It was only allowed to attach comments to lines the pull request changed. It was told not to report missing tests, because those comments used to annoy people. It reviewed with a single cautious persona that was explicitly instructed to prefer a few high confidence findings over speculation. It saw the diff and the changed files, but not the neighboring code, and not even the pull request title, so it reviewed every change without knowing what the change was for.
Each rule sounds sensible on its own, and together they quietly redefined the job. Think of a missing test, or a cleanup hook that should have been added and wasn't. Bugs like these are omissions, and an omission has no changed line to point at. Under a rule that says "only comment on changed lines", the reviewer cannot express the thought at all. Seven of the twelve problems the panel found were exactly this kind of bug.
There was one bug I used as a tracer. A hardcoded class name that should have been dynamic, sitting in plain sight in the diff. Across 25 recorded reviews spanning four different versions of the pipeline, that finding never showed up once, at any stage. The filters never got a chance to drop it because no reviewer pass ever wrote it down.
Which also explains why months of dashboards never showed me the problem. My metrics counted findings that got filtered, findings that got posted, findings that got disputed. A finding the model never wrote down is not in any of those counts. From the outside, silence and correctness look identical.
The fixes that made it worse
Before the rebuild, I tried the reasonable things. I want to list these honestly, because every one of them felt obviously correct before I measured it.
Give it the full files. If the reviewer misses things because it only sees the diff, show it every changed file in full. Recall, the share of real bugs the reviewer actually catches, got worse. On one test pull request, a single auto-generated translation file made up 97 percent of what I sent the model, and the two bugs I knew were in there went from partially caught to not caught at all. The model just drowned in it.
Trim the context, quietly. I capped the size and silently dropped what didn't fit. The model noticed a symbol it couldn't verify, invented a problem about it, and blocked a perfectly clean pull request. Partial context turned out to be worse than none.
Show it the callers. Bugs often live in the relationship between the diff and code elsewhere in the repo, so I injected the places each changed symbol gets used. Against 18 real bugs I pulled from our git history, recall went down, from 12 to 11. Reading page after page of "here is where this is used" made the model relax. It treated heavy usage as evidence that somebody must have handled things. I started calling this false reassurance.
Demand the reviewer prove its claims. I added a strict judge that kept a finding only if it could point at verifiable evidence. Recall fell from 55.6 percent to 46.3 percent. The judge did reduce randomness, exactly as designed. It stabilized borderline findings into consistent misses.
Let it explore the repo like an agent. I built an open-book reviewer that could search and read the codebase mid-review, the shape a lot of code review tools are adopting. In its first production trial it ran out of budget on five out of five pull requests and returned nothing. After three rounds of surgery it managed 18 shadow runs that produced exactly one finding the main pipeline had missed. On inspection, that one finding was wrong.
Five sensible ideas, four of them net negative and one a wash. Without the measurements I would have shipped all five and congratulated myself.
The rethink: argue first, verify once
The June panel pointed somewhere different. It had no smarter model and no secret context. What it had was a division of labor. Each reviewer on it only had to come up with suspicions, and a separate editing pass decided which suspicions were true. Mine had been making a single model do both jobs at once, and the caution the second job needs had been strangling the first.
So I rebuilt the reviewer in that shape. Three words to keep track of, because I'll use them from here on. A lens is one narrow-angle reviewer; one lens might care only about security, another only about missing tests. The panel is the group of lenses reviewing side by side. The curator is the editor that checks everything the panel produces. Here is the whole flow:
A continuous loop: pull request context is consumed on the left, the lenses emit a noisy stream of suspicions, most are dropped at the curator gate with a cited reason, and the few survivors become posted review comments on the right.
Context in, comments out
01
gather context
diff, full changed files, neighbors, schema, PR intent
02
plan the lenses
one cheap call designs 4 to 6 angles for this PR
03
run the panel
every lens reviews in parallel: miss nothing
04
curate
merge duplicates, drop only with cited evidence, rank
05
post
decide what is worth posting and where it goes
The old system asked one cautious reviewer to be right on the first pass, so every safety rule ran before or during the moment findings were generated. The new one lets four to six deliberately brave reviewers over-report, then pays for precision exactly once, at the end, where a wrong idea can be dropped without ever reaching a human. The lenses are allowed to flag unchanged lines and things that are missing entirely. The curator may only kill a finding by citing evidence: a specific line, a quote from the code. "I doubt it" is not grounds for deletion. If it cannot prove a finding wrong, the most it can do is downgrade it to a minor note instead of deleting it.
My favorite detail from all the measuring is that the cheap, fast model made a better panelist than the expensive one. As a lens, Gemini Flash came up with 20 to 27 suspicions per review; Gemini Pro, given the same job, offered 8 to 12. The expensive model was too dignified to guess. So the cheap model does the arguing, and the expensive one decides what survives.
Then I replayed the two pull requests from June through the rebuilt pipeline, with nothing about those PRs baked into any prompt. The panel turned up 7 of the 12 problems. The old reviewer, remember, had found 1 of the 12. I also pointed it at the 18 real bugs from our git history, the same set that sank the earlier fixes, and on the portion I had never tuned against it flagged 71.4 percent where the old reviewer managed 14.3. Same pattern both times.
Running it against the old reviewer
Replays can only tell you so much, though, so the next step was production traffic. I shipped the panel behind a flag with a shadow mode: on every NeetoCal pull request, the old reviewer posts its review as usual, and the panel runs silently alongside, recording what it would have said. NeetoCal is our scheduling product and one of our busiest repositories, which makes it a good stress test.
The first six shadowed pull requests told the whole story in miniature. The old reviewer posted "clean, nothing to flag" on all six. Internally it had not even produced a suspicion to dismiss. The panel disagreed on three of the six, and I hand-checked each disagreement against the code before believing it. All three held up. One was a customer-facing bug in a brand new feature, and another was caught while its pull request was still open, early enough to be fixed before merge.
Across six shadowed pull requests, the old reviewer called all six clean; the panel found three real bugs, agreed two were clean, and crashed once.
Clean from one reviewer, mixed from the panel
The quiet half matters just as much. The pull requests the panel called clean really were clean, and inside every review the curator was deleting the panel's bad guesses before they went anywhere. On one dependency update, a few lenses claimed version changes that were not in the diff at all, and the curator threw each claim out by pointing at exactly that. Which is the behavior I split the jobs to get.
One run also failed, which is what shadow mode is for. The curator has a budget of repository lookups, and on one review it hit the ceiling and gave up, twice, spending 54 cents to say nothing. I changed what hitting the budget means. Instead of crashing, the curator now has to make its decisions with whatever evidence it has already gathered.
Each shadow review cost between 15 and 58 cents and took about two minutes. The panel now reviews NeetoCal for real, and the old reviewer only steps in as a backup if a panel run fails. The other 200 repositories follow only if the data keeps agreeing.
The money question
The panel costs roughly 30 to 50 cents per review, where the old pipeline averaged about 7. Once every repository is on it, the monthly bill goes from about 125 dollars to somewhere between 700 and 1,000. A five to eight times bigger bill is the kind of number that needs defending, so let me defend it.
When my CEO saw the projection, his reaction was that even 1,000 dollars a month is cheap, and his arithmetic is hard to argue with. A single real bug that reaches customers costs more than that in support hours, fixes, and trust, and the shadow data shows the panel catching exactly that kind of bug on PRs the old system approved.
The other comparison is what everyone else charges. The established code review products price per seat: 24 dollars per user per month for one, 30 dollars per seat plus 1 dollar per review beyond a quota for another, and the tool NeetoBugWatch itself replaced cost 15 dollars per user per month. Neeto has close to a hundred developers.
Per-seat pricing at our size lands between 1,500 and 2,900 dollars a month, and the per-review overage model would cross 2,300 dollars on our volume alone. The panel at its most expensive is still cheaper than the cheapest of those.
The part I like most is not even the total. Per-seat tools charge for every developer whether they open a pull request or not. Neeto pays per review actually run, and because the pipeline is ours, I choose what a review is allowed to cost.
What I would tell you to steal
- Trace your misses, not your averages. My dashboards looked fine for months. Following individual missed bugs through the pipeline to see where each one got lost taught me more in an afternoon than the metrics had all quarter.
- Anything that suppresses output before it exists will never show up in your logs. A bad comment can be deleted later. A comment the model was never allowed to write leaves nothing behind to count.
- Trust your negative results. Full files, neighbor context, a stricter judge, an exploring agent: each made recall worse. Knowing four tempting approaches are dead ends is real progress, even though no metric moved when I learned it.
- More context can mean less recall. Models read bulk usage context as reassurance. Only add context that can settle a specific question.
- Make the cheap model brave and the expensive model careful. Generating suspicions needs volume and nerve, which the cheap model has. Judging them needs care, which is what the expensive one is actually worth paying for.
- Make dropping a finding cost as much as raising one. The curator has to cite code to kill a finding. Before that rule, it deleted real bugs on nothing more than its own confident say-so.
- Shadow before belief. Every conviction here was a hypothesis until the panel ran beside the old reviewer on real traffic and I verified the disagreements myself.
That June re-review found twelve problems because several narrow reviewers argued from different angles and one careful editor settled the argument. The rebuilt NeetoBugWatch is that June panel with my part automated away. It assembles its own reviewers, argues with itself, settles the argument, and posts the result without me in the room. For now the panel reviews NeetoCal and nothing else, and I am reading every verdict it posts, because blind faith in a code reviewer is exactly what got me here. The other 200 repositories get it when its probation ends, and its probation ends when I say so.