SciPaths: Forecasting Pathways to Scientific Discovery

TL;DR

SciPaths introduces a new benchmark for forecasting scientific discovery pathways, emphasizing the importance of understanding dependencies and prior work in scientific progress.

Contribution

The paper presents SciPaths, a novel benchmark dataset with expert-annotated pathways to evaluate AI models' ability to identify enabling scientific contributions.

Findings

Best language model achieves only 0.189 F1 score in pathway recovery.

Grounding prior work improves significantly with gold enabling contributions.

Decomposition quality is a key bottleneck for end-to-end scientific pathway prediction.

Abstract

Scientific progress depends on sequences of enabling contributions, yet existing AI4Science benchmarks largely focus on citation prediction, literature retrieval, or idea generation rather than the dependencies that make progress possible. In this paper, we introduce discovery pathway forecasting: given a target scientific contribution and the prior literature available at a specified time, the task is to (1) identify the enabling contributions required to realize it and (2) ground each in prior work when such prior work exists. We present SciPaths, a benchmark of 262 expert-annotated gold pathways and 2,444 silver pathways constructed from machine learning and natural language processing papers, where each pathway records enabling contributions, roles, rationales, and prior-work groundings or unmapped decisions. Evaluating frontier and open-weight language models, we find that the best model reaches only 0.189 F1 under strict semantic matching, with core methodological dependencies hardest to recover. Prior-work grounding improves substantially when gold enabling contributions are provided, showing that decomposition quality is a major bottleneck for end-to-end pathway recovery. SciPaths therefore shifts evaluation toward a missing capability in scientific forecasting: reasoning backward from a target contribution to the enabling scientific building blocks and prior-work dependencies that make it feasible.