ResearchPapers, evidence & method
Ai2 benchmark finds GPT and Claude novelty judges reverse verdicts after small prompt changes
Ai2's new benchmark shows GPT and Claude models judging research-idea novelty swing sharply with minor prompt edits, especially on AI-generated ideas. Retrieval and extra reasoning did not reliably help, and a simply prompted Claude Opus 4.6 outperformed costlier dedicated novelty checkers.
From the Hugging Face article of October 27, 2026 and cited sources: Changing a few words in the prompt moved GPT-5.4's accuracy at picking the more novel idea, when comparing human-written and LLM-generated ideas, from 93.5% to 40.9%. [1] [8]
SOURCES[1] Can we trust LLMs to judge whether a research idea is novel? huggingface.co[8] full text v3 arxiv.org
AT Pulse research deskPublished · Updated · 5 min read · AI-assisted analysisAI-assisted, not human-edited. Methodology Corrections Who owns this site
- What happened
- Changing a few words in the prompt moved GPT-5.4's accuracy at picking the more novel idea, when comparing human-written and LLM-generated ideas, from 93.5% to 40.9%. [1] [8]
- Original source
- Hugging Face article, company source
- Brief by
- AT Pulse research desk AI-assisted; publisher and contact on the desk page
- When
- October 27, 2026 article published
- Covered
- Oct 29 edition
- Evidence
- 9 sources across 4 sites
Reach: not measured yet
Changing a few words in the prompt moved GPT-5.4's accuracy at picking the more novel idea, when comparing human-written and LLM-generated ideas, from 93.5% to 40.9%. It reversed more than half its verdicts. On human-only pairs the same change shifted accuracy by about 7 points for GPT-5.4 and 21 points for Opus 4.6. Rewriting ideas into a common format did not remove the instability. [1] [8]
No single prompt worked best: the same edit helped some judges and hurt others. When judges compared ideas from two different generator models, roughly a quarter to a third of verdicts changed across prompt variants. That makes single-judge, single-prompt rankings of ideation systems fragile. [1] [8]
Extra machinery did not reliably help. Standard retrieval added at most 16 points, and only for some models. An agentic GPT-5.6 Sol judge with arXiv search scored 0.78 macro-F1 vs. 0.89 for Opus 4.6 at about 10× the cost. Prompted Opus 4.6 scored 0.854 for $2.59, vs. 0.642 for Idea Novelty Checker ($30.66) and 0.772 for AI Scientist's evaluator ($82.05). All of these are the authors' own results, and Ai2 co-authored Idea Novelty Checker. [8] [4] [2]
novelty verdicts depend heavily on prompt wording, especially on generated ideas. Implication: claims that a system produces more novel ideas, when scored by one LLM judge and prompt, may reflect setup choices.
Read the full assessment
Testing across several prompts and judges, with human review, is warranted.
Executive brief
Changing a few words in a prompt moved GPT-5.4's accuracy at picking the more novel of two research ideas from 93.5% to 40.9%, which is below chance. Ai2's study "Old Ideas, Novel Problems" tested six GPT and Claude judges. A plain prompted Claude Opus 4.6 beat two purpose-built novelty checkers at a fraction of the cost.
Read the full section
Changing a few words in a prompt moved GPT-5.4's accuracy at picking the more novel of two research ideas from 93.5% to 40.9%, which is below chance. On those same pairs, the model reversed more than half of its verdicts (Ai2 blog). Ai2's study "Old Ideas, Novel Problems" tested six GPT and Claude judges. Its central finding is that LLM "novelty judges" are unstable, and the instability is worst on AI-generated ideas, which are the ideas these judges are increasingly used to score. Adding retrieval or more reasoning did not reliably help. A plain prompted Claude Opus 4.6 beat two purpose-built novelty checkers at a fraction of the cost. No independent replication has been published yet.
What changed and event timeline
Sakana's AI Scientist ships an automated novelty check
The end-to-end research agent includes an LLM step that searches the literature for novelty. That step is one of the two dedicated evaluators Ai2 later benchmarked ().
LLM ideas judged more novel than human ones
In a study of 100+ NLP researchers, expert reviewers rated LLM-generated ideas as more novel than human ideas (p < 0.05) and slightly weaker on feasibility ().
Idea Novelty Checker proposed
Ai2-affiliated authors released a retrieval and reranking pipeline. They reported about 13% higher agreement with experts than earlier approaches ().
RINoBench shows poor judgment accuracy
This 1,381-idea benchmark found the best macro-F1 was 17.2. The judges' written justifications resembled expert reasoning, but their scores did not match expert labels ().
Ai2 paper posted
Sternlicht, Shahid, Jansen, Weld, Siangliulue and Hope posted v1. The current version, v3, is dated Oct 26 ().
Capabilities and access
- Judges tested: GPT-5.1, GPT-5.2, GPT-5.4, Claude Sonnet 4.5, Claude Opus 4.5 and Claude Opus 4.6. An agentic GPT-5.6 Sol variant used arXiv search (Ai2 blog).
- Release: two evaluation settings, each in pointwise (score one idea) and pairwise (compare two) formats, plus a pipeline that mines OpenReview reviews to build new sets. Code is Apache 2.0 and data is ODC-BY 1.0.
Read the full section
- Judges tested: GPT-5.1, GPT-5.2, GPT-5.4, Claude Sonnet 4.5, Claude Opus 4.5 and Claude Opus 4.6. An agentic GPT-5.6 Sol variant used arXiv search (Ai2 blog).
- Release: two evaluation settings, each in pointwise (score one idea) and pairwise (compare two) formats, plus a pipeline that mines OpenReview reviews to build new sets. Code is Apache 2.0 and data is ODC-BY 1.0. Running it needs OpenAI, Anthropic and Semantic Scholar API keys (GitHub).
Technical analysis for researchers and developers
- How labels were built: the pipeline pulled reviewer passages that explicitly praise or dispute originality, rather than using paper acceptance as a proxy.
- Human-Only setting: 154 high-novelty ideas from top-10% accepted ICLR 2026 papers vs. 145 lower-novelty ideas from bottom-10% rejected papers.
- Human+Generated setting: the same 154 human ideas vs. 154 ideas from a single-pass LLM generator with no tools.
Read the full section
- How labels were built: the pipeline pulled reviewer passages that explicitly praise or dispute originality, rather than using paper acceptance as a proxy.
- Human-Only setting: 154 high-novelty ideas from top-10% accepted ICLR 2026 papers vs. 145 lower-novelty ideas from bottom-10% rejected papers.
- Human+Generated setting: the same 154 human ideas vs. 154 ideas from a single-pass LLM generator with no tools.
- Label checks: extraction precision was 94%, with κ=0.89 on 50 samples (paper).
- Protocol: one variable changed at a time. Pairwise calls ran three times per presentation order to control for position bias.
- Style control: rewriting all ideas into a common purpose/mechanism format did not remove the instability. Some changes still flipped more than two-thirds of verdicts.
- Cost: about $15,000 in API spend.
Claims and evidence
- Prompt sensitivity: a prompt change swung accuracy by more than 50 points on Human+Generated.
- No universally best prompt: the same prompt edit helped some judges and hurt others.
- Comparing two generators: when judges compared ideas from two different generator models, roughly a quarter to a third of verdicts changed across prompt variants.
Read the full section
- Prompt sensitivity: a prompt change swung accuracy by more than 50 points on Human+Generated. The same change moved it by about 7 points on Human-Only for GPT-5.4 and by 21 points for Opus 4.6 (Ai2 blog).
- No universally best prompt: the same prompt edit helped some judges and hurt others.
- Comparing two generators: when judges compared ideas from two different generator models, roughly a quarter to a third of verdicts changed across prompt variants.
- Retrieval and reasoning: standard retrieval added at most 16 points, and only for some models. The agentic judge scored 0.78 macro-F1 vs. 0.89 for Opus 4.6, at about 10× the cost (paper).
- Dedicated tools: prompted Opus 4.6 scored 0.854 for $2.59. Idea Novelty Checker scored 0.642 for $30.66 and AI Scientist's evaluator 0.772 for $82.05.
- How to read these numbers: they are all the authors' own results. Ai2 co-authored Idea Novelty Checker, so it is grading its own earlier tool. Press coverage such as HyperAI restates Ai2's findings and adds no independent testing.
Context and prior work
- Positioning: the work fits a growing body of evidence that LLM novelty judgment is weak. Earlier, RINoBench found judges that sound plausible but score inaccurately.
- Tension with earlier findings: human reviewers in Si et al. rated LLM ideas as more novel.
- Ai2's motivation: its own discovery tools need a reliable novelty filter. These include DataVoyager, AutoDiscovery, Theorizer and Asta (Ai2 blog).
Read the full section
- Positioning: the work fits a growing body of evidence that LLM novelty judgment is weak. Earlier, RINoBench found judges that sound plausible but score inaccurately.
- Tension with earlier findings: human reviewers in Si et al. rated LLM ideas as more novel. In Ai2's study, an expert preferred the human idea in all 30 audited pairs. The two studies used different generators, baselines and human comparison pools, so the results are not directly comparable.
- Ai2's motivation: its own discovery tools need a reliable novelty filter. These include DataVoyager, AutoDiscovery, Theorizer and Asta (Ai2 blog).
Limitations, safety and contested findings
- Weak labels: the Human+Generated labels assume that ideas from top-decile ICLR papers beat single-pass LLM ideas.
- Partial coverage: testing one factor at a time leaves interactions between factors unexplored.
- Scope: the study covers novelty only, not impact or surprise, and draws only on ICLR/ML venues.
Read the full section
- Weak labels: the Human+Generated labels assume that ideas from top-decile ICLR papers beat single-pass LLM ideas. Only 30 pairs were checked by an expert, and that sample was weighted toward cases the judge got wrong (paper).
- Partial coverage: testing one factor at a time leaves interactions between factors unexplored.
- Scope: the study covers novelty only, not impact or surprise, and draws only on ICLR/ML venues.
- Conservative design: the generator was deliberately basic. How well judges handle stronger, agentic ideation systems has not been tested.
- Ties: Claude Sonnet 4.5 returned ties on more than half of pairwise judgments in some configurations.
Business and practitioner implications
- Treat novelty claims skeptically: a vendor's "our system generates more novel ideas" claim based on an LLM judge may mostly reflect how the judge was prompted.
- Validate on the right inputs: test judges on generated ideas, not only human-written papers.
- Start cheap: a simple prompted judge with majority voting is a strong, inexpensive baseline.
Read the full section
- Treat novelty claims skeptically: a vendor's "our system generates more novel ideas" claim based on an LLM judge may mostly reflect how the judge was prompted. Ask for results across several prompts and several judge models.
- Validate on the right inputs: test judges on generated ideas, not only human-written papers.
- Start cheap: a simple prompted judge with majority voting is a strong, inexpensive baseline. Expensive agentic or retrieval-based pipelines need to show they actually beat it.
- Budget for checks: set aside spending for sensitivity analysis and keep humans reviewing novelty decisions that matter.
The source trail.
Sources (9)
Can we trust LLMs to judge whether a research idea is novel?
Article text retrieved; extracted text may omit tables or interactive elements.
huggingface.co