SciImpact: A Multi-Dimensional, Multi-Field Benchmark for Scientific Impact Prediction

TL;DR

SciImpact introduces a comprehensive benchmark for predicting scientific impact across multiple dimensions and fields, evaluating various large language models and highlighting the benefits of multi-task fine-tuning.

Contribution

It presents a large-scale, multi-dimensional benchmark for scientific impact prediction, integrating diverse data sources and evaluating LLMs across 19 fields.

Findings

Multi-task fine-tuning improves smaller LLMs performance.

Off-the-shelf models show high variability across impact dimensions.

Fine-tuned smaller models can outperform larger, closed-source models.

Abstract

The rapid growth of scientific literature calls for automated methods to assess and predict research impact. Prior work has largely focused on citation-based metrics, leaving limited evaluation of models' capability to reason about other impact dimensions. To this end, we introduce SciImpact, a large-scale, multi-dimensional benchmark for scientific impact prediction spanning 19 fields. SciImpact captures various forms of scientific influence, ranging from citation counts to award recognition, media attention, patent reference, and artifact adoption, by integrating heterogeneous data sources and targeted web crawling. It comprises 215,928 contrastive paper pairs reflecting meaningful impact differences in both short-term (e.g., Best Paper Award) and long-term settings (e.g., Nobel Prize). We evaluate 11 widely used large language models (LLMs) on SciImpact. Results show that off-the-shelf models exhibit substantial variability across dimensions and fields, while multi-task supervised fine-tuning consistently enables smaller LLMs (e.g., 4B) to markedly outperform much larger models (e.g., 30B) and surpass powerful closed-source LLMs (e.g., o4-mini). These results establish SciImpact as a challenging benchmark and demonstrate its value for multi-dimensional, multi-field scientific impact prediction. Our project homepage is https://flypig23.github.io/sciimpact-homepage/