Is this Idea Novel? An Automated Benchmark for Judgment of Research Ideas

TL;DR

This paper introduces RINoBench, a comprehensive benchmark for evaluating automated research idea novelty judgments, revealing that current large language models struggle to match human assessments despite mimicking reasoning.

Contribution

It presents the first large-scale, standardized benchmark for research idea novelty evaluation, enabling consistent comparison of automated methods against human judgments.

Findings

LLMs' reasoning aligns with humans but not accuracy

Automated models often diverge from human novelty judgments

Benchmark facilitates large-scale, standardized evaluation of novelty assessment methods

Abstract

Judging the novelty of research ideas is crucial for advancing science, enabling the identification of unexplored directions, and ensuring contributions meaningfully extend existing knowledge rather than reiterate minor variations. However, given the exponential growth of scientific literature, manually judging the novelty of research ideas through literature reviews is labor-intensive, subjective, and infeasible at scale. Therefore, recent efforts have proposed automated approaches for research idea novelty judgment. Yet, evaluation of these approaches remains largely inconsistent and is typically based on non-standardized human evaluations, hindering large-scale, comparable evaluations. To address this, we introduce RINoBench, the first comprehensive benchmark for large-scale evaluation of research idea novelty judgments. It comprises 1,381 research ideas derived from and judged by human experts as well as nine automated evaluation metrics designed to assess both rubric-based novelty scores and textual justifications of novelty judgments. Using this benchmark, we evaluate several state-of-the-art large language models (LLMs) on their ability to judge the novelty of research ideas. Our findings reveal that while LLM-generated reasoning closely mirrors human rationales, this alignment does not reliably translate into accurate novelty judgments, which diverge significantly from human gold standard judgments - even among leading reasoning-capable models. Data and code available at: https://github.com/TimSchopf/RINoBench.