PreScience: A Benchmark for Forecasting Scientific Contributions

TL;DR

PreScience introduces a comprehensive benchmark for forecasting scientific progress using AI, involving four interconnected tasks and a large dataset of AI research papers to evaluate predictive capabilities and generate insights into future scientific trends.

Contribution

The paper presents a new benchmark dataset and tasks for scientific forecasting, along with baseline models and a novel contribution similarity metric, advancing research in predictive science.

Findings

Significant room for improvement in contribution prediction accuracy.

Current LLMs only moderately match ground-truth contributions.

Synthetic scientific corpora are less diverse and novel than human research.

Abstract

Can AI systems trained on the scientific record up to a fixed point in time forecast the scientific advances that follow? Such a capability could help researchers identify collaborators and impactful research directions, and anticipate which problems and methods will become central next. We introduce PreScience -- a scientific forecasting benchmark that decomposes the research process into four interdependent generative tasks: collaborator prediction, prior work selection, contribution generation, and impact prediction. PreScience is a carefully curated dataset of 98K recent AI-related research papers, featuring disambiguated author identities, temporally aligned scholarly metadata, and a structured graph of companion author publication histories and citations spanning 502K total papers. We develop baselines and evaluations for each task, including LACERScore, a novel LLM-based measure of contribution similarity that outperforms previous metrics and approximates inter-annotator agreement. We find substantial headroom remains in each task -- e.g. in contribution generation, frontier LLMs achieve only moderate similarity to the ground-truth (GPT-5, averages 5.6 on a 1-10 scale). When composed into a 12-month end-to-end simulation of scientific production, the resulting synthetic corpus is systematically less diverse and less novel than human-authored research from the same period.