Good Idea or Not, Representation of LLM Could Tell

TL;DR

This paper explores using large language model representations to evaluate scientific ideas, introducing a benchmark dataset and a framework that correlates well with human judgment, advancing automated idea assessment.

Contribution

It presents a new benchmark dataset and a framework leveraging LLM representations for quantitative idea evaluation, showing potential beyond generative outputs.

Findings

LLM representations correlate with human idea assessments

The proposed method outperforms baseline approaches

Representations are more effective than generative outputs for this task

Abstract

In the ever-expanding landscape of academic research, the proliferation of ideas presents a significant challenge for researchers: discerning valuable ideas from the less impactful ones. The ability to efficiently evaluate the potential of these ideas is crucial for the advancement of science and paper review. In this work, we focus on idea assessment, which aims to leverage the knowledge of large language models to assess the merit of scientific ideas. First, we investigate existing text evaluation research and define the problem of quantitative evaluation of ideas. Second, we curate and release a benchmark dataset from nearly four thousand manuscript papers with full texts, meticulously designed to train and evaluate the performance of different approaches to this task. Third, we establish a framework for quantifying the value of ideas by employing representations in a specific layer of large language models. Experimental results show that the scores predicted by our method are relatively consistent with those of humans. Our findings suggest that the representations of large language models hold more potential in quantifying the value of ideas than their generative outputs, demonstrating a promising avenue for automating the idea assessment process.