ForeCite: Adapting Pre-Trained Language Models to Predict Future Citation Rates of Academic Papers

TL;DR

ForeCite leverages pre-trained causal language models with a linear head to accurately predict future citation rates of academic papers, significantly outperforming previous methods and demonstrating robustness across various model sizes and datasets.

Contribution

The paper introduces ForeCite, a novel framework that adapts pre-trained causal language models for citation rate prediction, achieving state-of-the-art performance.

Findings

Achieves a test correlation of 0.826 on biomedical papers.

Outperforms previous state-of-the-art by 27 points.

Demonstrates consistent gains across model sizes and data volumes.

Abstract

Predicting the future citation rates of academic papers is an important step toward the automation of research evaluation and the acceleration of scientific progress. We present $\textbf{ForeCite}$, a simple but powerful framework to append pre-trained causal language models with a linear head for average monthly citation rate prediction. Adapting transformers for regression tasks, ForeCite achieves a test correlation of $\rho = 0.826$ on a curated dataset of 900K+ biomedical papers published between 2000 and 2024, a 27-point improvement over the previous state-of-the-art. Comprehensive scaling-law analysis reveals consistent gains across model sizes and data volumes, while temporal holdout experiments confirm practical robustness. Gradient-based saliency heatmaps suggest a potentially undue reliance on titles and abstract texts. These results establish a new state-of-the-art in forecasting the long-term influence of academic research and lay the groundwork for the automated, high-fidelity evaluation of scientific contributions.