Code review benchmark
AI code review benchmarks
Greptile's public benchmark puts AI code reviewers on 50 real bugs from five production codebases. We ran Aether Review on the same pull requests with its stock settings, graded Aether's reviews and Greptile's own by hand under Greptile's published rule, and ran Martian's judge for precision. Every review and every grading decision is public.
42 of 50 known bugs caught
Graded the same way, Greptile caught 35. Its own page claims 41.
- Critical
- 11/15
- High
- 15/17
- Medium
- 9/11
- Low
- 7/7
Catch rate by tool
Share of the 50 known bugs each tool caught with a line-level review comment.
- 84%
- 70%
- 58%
- 52%
- 44%
- 6%
- Aether42/50
- Greptile35/50
- Bugbot29/50
- Copilot26/50
- CodeRabbit22/50
- Graphite3/50
- Tools
- 6
- Real bugs
- 50
- Repositories
- 5
- Languages
- 5
Overview
Fifty real bugs. Six reviewers.
Greptile published a benchmark of five AI code review tools on 50 pull requests from open-source codebases. Each PR is a real change that introduced a bug a later commit fixed, re-opened as a fresh pull request. We ran Aether Review on the same 50 PRs with its stock product settings.
The score asks one question: did the tool find the bug? A catch is an inline comment on the faulty change that states the defect and gets its mechanism right. Naming the line but calling the code safe, or claiming a different problem, does not count. False positives, style suggestions and unrelated comments do not move the catch rate, so this page also reports precision under Martian's judge.
Every review is a public pull request, so you can open any square in the matrix or any mark in the case library and judge it yourself. The grading decisions, with a flag on every judgment call, are on GitHub.
Bugs caught by severity
Severity as labeled in Greptile's published table.
- Aether
- Greptile
- Greptile's own claim
- 73%53%
- 88%76%
- 82%91%
- 100%57%
- Critical15 bugs
- High17 bugs
- Medium11 bugs
- Low7 bugs
Methodology
How the benchmark runs.
The dataset is Greptile's: five open-source repositories in five languages, ten bug-fix PRs from each, traced back to the commits that introduced the bugs. Each case is a fresh PR that re-opens the original change on a clean fork with full repository access; for the 35 cases keyed to an upstream pull request, the benchmark PR has that PR's own base and head commits, so the diff under review is the original change.
Aether Review ran on 50 fresh forks of the same PRs with the product's default deep review prompt and Codex gpt-6-astra at high reasoning, one posted review per fork, from October 13, 01:09 UTC, to October 14, 00:53 UTC. A review builds a one-hop context pack for every hunk, runs stock static analyzers with their default rule sets for leads, reads each hunk with six independent reviewer passes and one auditor, has a file owner read each changed file whole, then merges the findings by root cause into one posted review of at most 30 inline comments.
A pull request counts as caught when its posted review has an inline comment on the faulty change that states the defect and gets its mechanism right. We also tried a lenient rule that credits a comment explaining the mechanism but calling the code safe; it changed no PR in any reported run. Both tools were graded by the same hand, and any call another grader could reasonably reject carries a flag in the decision file.
Greptile's side is its bot's public comments on the benchmark forks, read on October 19, 2026 and graded under the same rule. Bugbot, Copilot, CodeRabbit and Graphite are shown as Greptile graded them in July 2025.
Test dataset
Ten bug-fix pull requests from each repository.
Caveats
- No holdout
- Twenty-two prompt versions were developed against these 50 PRs. Each reported run is fresh forks of a text frozen before the run, but the text was shaped by earlier runs on the same PRs, and some generic rules were added after specific misses. Treat the numbers as in-sample.
- Precision
- The catch rate does not measure precision. The 50 reviews posted 737 inline comments (35 P1, 320 P2, 382 nits), a median of 13 per review, about three times Greptile's. Under Martian's judge, which scores every comment against 173 golden comments, Aether's recall is 0.77 and its precision 0.14, the highest and the lowest of the six tools judged: 772 comments match no golden comment, about 15 per PR. A code check of sixty of those found most high-agreement ones to be real defects outside the golden set, and most single-reviewer ones minor or wrong.
- Judgment calls
- 11 of Aether's 42 catches are flagged in the decision file as calls another grader could reject, mostly comments anchored a call away from the faulty line or framing the defect differently from the golden comment. Without them the count is 31.
- Disputed targets
- On 5 PRs Greptile's one-line description of the bug matches neither its own fork nor Martian's copy. We score the defect the code has instead; Aether catches 4 of those. Counting all 5 as misses gives 38.
- One grader
- Greptile's 41 is its grading of itself. Greptile's 35 and Aether's 42 are our grading, by the team that wrote Aether's prompts. Every decision file records which comment matched which golden comment, so each call is checkable.
- One run
- Variance is not measured. One same-text repeat of a single PR flipped from miss to catch, so at least one PR moves between runs of the same text. 8 reviews died in Codex usage-limit outages and were started again on the same PR; no PR received two completed reviews.
- Earlier scanners
- An earlier prompt also scored 42 with pre-pass scanners written while looking at these PRs; with the scanners removed and nothing else changed it scored 35. No reported run uses them.
- What ships
- Since 20 October the shipped tier also applies a publishing policy: inline only for findings with two or more independent sources or a confirmed verification, the rest in the summary. Simulated on this run's reviews it keeps 41 of 42 catches and trims the judged false positives; a fresh run has not measured it.
Results
Bug catch performance
Aether caught 42 of 50 known bugs, a catch rate of 84%, at about $29 of model cost per review. Graded the same way from its public reviews, Greptile caught 35 (70%); its own page claims 41 (82%). As Greptile graded them, Bugbot caught 58%, Copilot caught 52%, CodeRabbit caught 44%, and Graphite caught 6%. The two tools miss different PRs: both catch 30, Aether alone 12, Greptile alone 5, neither 3.
Every bug, every tool
One square per bug, grouped by repository. A filled square is a catch. Click any square to open that review.
- Caught
- Missed
- Claimed by Greptile, not credited
Critical
- 73%
- 53%
- 60%
- 53%
- 33%
- 7%
- Aether
- Greptile
- Bugbot
- Copilot
- CodeRabbit
- Graphite
High
- 88%
- 76%
- 53%
- 41%
- 35%
- 0%
- Aether
- Greptile
- Bugbot
- Copilot
- CodeRabbit
- Graphite
Medium and low
- 89%
- 78%
- 61%
- 61%
- 61%
- 11%
- Aether
- Greptile
- Bugbot
- Copilot
- CodeRabbit
- Graphite
Case library
Every review, in public.
Every PR in the test set, with Greptile's one-line summary of the bug, its severity, and whether each tool caught it. Tool names link to each tool's run, and every mark links to the exact PR so you can read the review yourself.
Caught means an inline comment on the faulty change that states the defect and gets its mechanism right. Naming the line but calling the code safe, claiming a different problem, or a summary-only mention does not count.
- Our grade of Greptile: Greptile's comment says OptimizedCursorPaginator does not exist. The PR defines it in paginator.py, so the flagged defect is not real, and the negative-slice bug is not mentioned.
- Aether's review: No comment on the line where the missing github_authenticated_user state is dereferenced.
- Aether's review: Neither the detector_type read nor the response-type change was commented on in any reported run.
- Aether's review: Two comments mention ShutDown, about pool reuse after close and draining before stop; neither names the second handler that exits the worker.
- Our grade of Greptile: Greptile's comment says queue.ShutDown does not exist. The fork pins Python 3.13, where it does, and the second handler is not mentioned.
- HighEnhanced Pagination Performance for High-Volume Audit LogsImporting non-existent OptimizedCursorPaginator
- Our grade of Greptile: Greptile's comment says OptimizedCursorPaginator does not exist. The PR defines it in paginator.py, so the flagged defect is not real, and the negative-slice bug is not mentioned.
- CriticalOptimize spans buffer insertion with eviction during insertNegative offset cursor manipulation bypasses pagination boundaries
- LowSupport upsampled error count with performance optimizationssample_rate = 0.0 is falsy and skipped
- CriticalGitHub OAuth Security EnhancementNull reference if github_authenticated_user state is missing
- Aether's review: No comment on the line where the missing github_authenticated_user state is dereferenced.
- CriticalReplays Self-Serve Bulk Delete SystemBreaking changes in error response format
- Aether's review: Neither the detector_type read nor the response-type change was commented on in any reported run.
- MediumSpan Buffer Multiprocess Enhancement with Health MonitoringInconsistent metric tagging with 'shard' and 'shards'
- MediumImplement cross-system issue synchronizationShared mutable default in dataclass timestamp
- HighReorganize incident creation / issue occurrence logicUsing stale config variable instead of updated one
- HighAdd ability to use queues to manage parallelismInvalid queue.ShutDown exception handling
- Aether's review: Two comments mention ShutDown, about pool reuse after close and draining before stop; neither names the second handler that exits the worker.
- Our grade of Greptile: Greptile's comment says queue.ShutDown does not exist. The fork pins Python 3.13, where it does, and the second handler is not mentioned.
- HighAdd hook for producing occurrences from the stateful detectorIncomplete implementation (only contains pass)
- Aether7/10
- Greptile6/10claims 8
- Bugbot4/10
- Copilot4/10
- CodeRabbit3/10
- Graphite0/10
Catch it before it merges.
Connect a GitHub repo and Aether reviews your next pull request with the code running in a dev preview.