A coding agent finishes a 500-line change in the time it takes you to read the ticket. The change compiles and the tests pass. Now someone has to decide whether it is correct, and that someone is still you.
That is the shift of the last eighteen months. Writing code used to be the expensive step and review was the cheap check at the end. Agents reversed it. Anthropic reported in March that code output per engineer at the company had grown 200% in a year [1]. Nobody's review capacity grew 200%.
The industry's answer has been to point a model at the review queue too. We think that is right. But a reviewer, human or model, is only worth what it is measured to be worth, and most AI review today is evaluated by vibes.
What changed in 2025–2026
Three things happened at once.
Review became a native feature. GitHub made Copilot code review generally available to paid Copilot subscribers in April 2025 [2]. Cursor shipped Bugbot to review pull requests [3]. OpenAI's Codex reviews pull requests on request or automatically, and its documentation says it flags only P0 and P1 issues so comments stay focused [4]. Anthropic added /security-review and a GitHub Action to Claude Code in August 2025 [5], then launched a multi-agent Code Review product in March 2026 that dispatches several agents per PR and verifies candidate bugs to filter false positives [1]. It became generally available this month [6].
Vendors started publishing quality numbers. Anthropic's launch post says the share of its internal PRs receiving substantive review comments went from 16% to 54%, and that under 1% of findings are marked incorrect by its engineers [1]. Those are useful numbers. They are also numbers about one company's code, measured by the vendor, against its own engineers' judgments.
The research showed that passing tests is not the same as passing review. METR had maintainers of scikit-learn, Sphinx and pytest blind-review 296 AI-written patches that had already passed the SWE-bench Verified grader. Roughly half would not have been merged, and after normalizing against a baseline of real merged human patches, maintainer merge rates averaged about 24 points below the automated grader's [7]. The automated check and the owner's judgment measure different things. DORA's 2025 report found AI adoption now correlates with higher delivery throughput, but delivery instability continues to rise [8]. Faster authorship without better verification moves the cost downstream.
The noise problem is older than LLMs
Every reviewer bot runs into the same wall: comments nobody acts on.
The best field data predates LLMs. Google's Tricorder team measured "effective false positives", findings where the developer took no positive action, and held analyzers in code review to under 10%. If an analyzer's "Not useful" ratio went above 10%, the team disabled it until it improved [9]. The bar is low noise, not high recall, because one ignored comment teaches a developer to ignore the next one.
LLM reviewers inherit this. A six-week study of an LLM review assistant at Mozilla and Ubisoft, covering 587 patch reviews and about 1.6k generated comments, found reviewers accepted 8.1% and 7.2% of comments respectively, with functional comments accepted less often than refactoring ones [10]. That is not a failure of one tool. It is what review looks like when nobody tunes against the developer's actual verdict.
So the question for any review tool is not "does it find bugs?" Every tool finds some. The question is: at what precision, at what recall, and at what cost per real finding, on your code.
Our position: three numbers, owner's truth
A review tool should be judged by three measured quantities:
- Precision. Of the findings it raises, how many are real defects.
- Recall. Of the real defects in the change, how many it raises.
- Cost per true positive. Dollars (or local wall time) divided by real findings, not per review.
And those numbers mean something only when graded against ground truth that someone accountable owns. Not the model's confidence score. Not a second model's opinion of the first. Not a vendor's aggregate across other people's code.
The strongest counter-argument is that ground truth is expensive and models are getting good enough to grade each other. We take that seriously, and we use model labelers, but as a measured instrument, never as the ruler. The reason is self-preference: when the labeler is also one of the reviewers under test, its labels drift toward its own findings. A model's agreement with a human owner is itself a number you report, not an assumption you make.
How Cervello Verify is built to be measured
Cervello Verify is our Git-native review harness. It reviews uncommitted or branch changes and emits structured findings with evidence. It is written so the core hypothesis can be falsified: that an LLM combined with deterministic repository context, static checks and executable evidence finds more real defects, with fewer false positives, than LLM-only review.
The engine
The core is a Rust workspace of small crates behind one Engine facade: read-only git (non-read-only subcommands are unrepresentable in the type system), immutable content-addressed checkpoints, deterministic context retrieval, a provider trait, a reviewer with versioned prompts, and SQLite storage. Secrets, symlinks, binaries and oversized files are excluded from snapshots and prompts, and every exclusion is listed in the report.
Two surfaces sit on the same engine. The cervello CLI runs in any repo. cervello mcp is a stdio MCP server with review, findings, explain, verify, confirm and dismiss tools, packaged as a Claude Code plugin with a /verify command and an opt-in Stop hook that reviews the agent's change before the session ends. The hook is off by default and fails open: a review that errors never traps the agent.
Suspected is not verified
The design separates review intelligence from verification intelligence. The model proposes; it never proves. Every model finding enters as suspected_defect. A finding becomes verified_defect only when a generated reproduction test actually ran in a sandbox and failed on an assertion, and verification_gap when it could not run. Sandboxed reproduction is the newest layer and still being hardened; until it runs, a finding stays suspected. The model's self-reported confidence is stored and labelled as model-reported, and never treated as proof.
finding.v1
status verified_defect | suspected_defect | verification_gap
lifecycle open | stale | superseded | dismissed
location { path, start_line, end_line } # must exist in the checkpoint
category logic | boundary | concurrency | security | authorization |
api_contract | idempotency | transaction | error_handling | ...
confidence 0..1 # model-reported, never proof
evidence[] { kind: test|trace|check, command, exit_code, ... }
Findings whose locations don't exist in the checkpoint are dropped and counted as invalid_location, a tracked metric. A reviewer that cites lines that aren't there should pay for it in the report.
The eval harness
Evaluation is a first-class feature, not an afterthought. A Python harness, cervello-eval, runs cervello eval run for each case × provider × strategy × repeat and grades the output deterministically. There is no LLM-as-judge in the grader.
A case is a repository state before a change, the change itself (left uncommitted, exactly what a developer would see), grader-only ground truth, and an optional grader-only reproduction test. For historical bugs, only the pre-fix state is ever an input; the fixing commit can inform the truth but never reaches the reviewer.
Grading is a pure function:
# A finding matches a truth item iff:
# same path, line ranges overlap within ±3 lines,
# categories equal or compatible (e.g. idempotency~transaction).
# Matching is one-to-one, greedy by line distance:
# a duplicate finding on an already-matched bug is a false positive.
precision = sum(tp) / (sum(tp) + sum(fp)) # micro, over runs
recall = sum(tp) / sum(truth_items) # micro, over runs
The report adds false positives per case, the false-positive rate on clean-patch controls (changes with no defect at all), invalid-location rate, p50/p95 latency, tokens and cost. Precision and recall carry 95% confidence intervals from a case-level bootstrap. When two configurations' intervals overlap, the report says "insufficient evidence" and declares no winner. Re-running with the same inputs produces identical quality numbers; only latency and timestamps vary.
The current corpus has 24 cases across dev, val and heldout splits: 20 synthetic cases over six small-to-PR-sized Python codebases (payments, authorization, job queues, billing, fulfillment, a multi-tenant workspace API) and 4 historical bugs extracted from public open-source projects with their upstream advisories or issues. Five are clean controls. The larger synthetic defects are deliberately cross-file: the wrong line looks plausible alone and is wrong only given a definition, caller or rule elsewhere, mixed in among legitimate refactors. Prompt tuning may touch only dev and val; heldout runs require an explicit flag and are appended to a log.
Local and closed models, same ruler
The harness compares providers under matched token budgets and inference parameters: local models served by Ollama on a Mac mini class machine, closed models through API keys, and personal coding-CLI subscriptions for exploratory runs (excluded from reported results, since they offer no seed or temperature control). Cost is reported in the unit that matters for each: dollars for API models, API-equivalent dollars for subscriptions, wall time at $0 marginal cost for local.
Local hardware has real limits, and we record them instead of hand-waving. On a 16 GB M4 Mac mini, we measured qwen3:8b at roughly 21 tokens/s generation with a 24k context, found memory pressure was the bottleneck, and found that with thinking enabled the model spent its whole 4,096-token output budget reasoning and returned no answer. Those are the kinds of facts that decide whether "run it locally" is a product or a slogan.
We are not publishing comparative precision and recall results yet. When we do, they will come with the corpus hash, prompt version, model ids and confidence intervals, so anyone can tell a real difference from noise.
Who labels what
Real diffs don't come with a truth file. For those we gather adjudication labels: real, false positive, or unsure. A model labeler pre-labels findings as a triage aid. The repository owner's labels are the ground truth, and the model labeler is scored by its agreement with them. A model labeling its own findings is reported separately.
This is internal evaluation tooling. End users never see a labeling queue. They get two verbs, confirm and dismiss, whose job is to give them value directly, such as not being shown the same false alarm twice in a repository. We don't silently turn user clicks into benchmark truth.
What we believe
Authorship is no longer the constraint. Verification is. The teams that win the next few years won't be the ones generating the most code; they'll be the ones that can say, with numbers, how much of it they've actually checked.
A review tool that can't report its precision, recall and cost per real finding on your code is asking for trust it hasn't earned. We would rather earn it with method: deterministic grading, owner-held truth, intervals instead of leaderboards, and a model that is allowed to suspect but never to declare. The same principle runs through Cervello's PR agent: the agent proposes a fix with a regression test, and a human always approves the merge.
More at https://www.cervellolab.xyz.
References
- Anthropic, "Code Review for Claude Code" (March 9, 2026). https://claude.com/blog/code-review
- GitHub Changelog, "Copilot code review now generally available" (April 4, 2025). https://github.blog/changelog/2025-04-04-copilot-code-review-now-generally-available/
- Cursor, Bugbot documentation. https://cursor.com/docs/bugbot
- OpenAI, "Codex code review in GitHub" (documentation). https://learn.chatgpt.com/docs/third-party/github.md
- Anthropic, "Automate security reviews with Claude Code" (August 2025). https://www.anthropic.com/news/automate-security-reviews-with-claude-code
- Anthropic, "Code Review is now generally available" (October 2026). https://claude.com/resources/articles/code-review-is-now-generally-available
- METR, "Many SWE-bench-Passing PRs Would Not Be Merged into Main" (March 10, 2026). https://metr.org/notes/2026-03-10-many-swe-bench-passing-prs-would-not-be-merged-into-main/
- DORA, "2025 State of AI-assisted Software Development". https://dora.dev/dora-report-2025/ (summary: https://www.infoq.com/news/2025/09/dora-state-of-ai-in-dev-2025)
- Sadowski et al., "Lessons from Building Static Analysis Tools at Google," Communications of the ACM 61(4), 2018. https://research.google/pubs/lessons-from-building-static-analysis-tools-at-google/
- Olewicki et al., "Impact of LLM-based Review Comment Generation in Practice: A Mixed Open-/Closed-source User Study," arXiv:2411.07091. https://arxiv.org/abs/2411.07091