Are Large Language Models able to Predict Highly Cited Papers? Evidence from Statistical Publications

TL;DR

This study investigates whether large language models can predict highly cited statistical papers using early textual information, demonstrating stable performance and revealing topic patterns associated with high impact.

Contribution

The paper introduces a text-centered framework leveraging LLMs and structured prompts to predict long-term citation impact from early publication data in the field of statistics.

Findings

LLMs achieve stable, competitive prediction performance.

Predicted highly cited papers focus on topics like causal inference and deep learning.

The approach generalizes well over different publication periods.

Abstract

Predicting highly-cited papers is a long-standing challenge due to the complex interactions of research content, scholarly communities, and temporal dynamics. Recent advances in large language models (LLMs) raise the question of whether early-stage textual information can provide useful signals of long-term scientific impact. Focusing on statistical publications, we propose a flexible, text-centered framework that leverages LLMs and structured prompt design to predict highly cited papers. Specifically, we utilize information available at the time of publication, including titles, abstracts, keywords, and limited bibliographic metadata. Using a large corpus of statistical papers, we evaluate predictive performance across multiple publication periods and alternative definitions of highly cited papers. The proposed approach achieves stable and competitive performance relative to existing methods and demonstrates strong generalization over time. Textual analysis further reveals that papers predicted as highly cited concentrate on recurring topics such as causal inference and deep learning. To facilitate practical use of the proposed approach, we further develop a WeChat mini program, \textit{Stat Highly Cited Papers}, which provides an accessible interface for early-stage citation impact assessment. Overall, our results provide empirical evidence that LLMs can capture meaningful early signals of long-term citation impact, while also highlighting their limitations as tools for research impact assessment.