Proof of Time: A Benchmark for Evaluating Scientific Idea Judgments

TL;DR

PoT is a benchmarking framework that evaluates scientific idea judgments by linking them to future observable signals, enabling scalable, verifiable assessment of models' forecasting abilities in scientific research.

Contribution

Introduces PoT, a semi-verifiable benchmark linking scientific idea judgments to future signals, facilitating scalable evaluation of models and agents in scientific forecasting tasks.

Findings

Higher interaction budgets improve agent performance.

Tool use benefits are task-dependent.

PoT enables scalable, future-verifiable evaluation.

Abstract

Large language models are increasingly being used to assess and forecast research ideas, yet we lack scalable ways to evaluate the quality of models' judgments about these scientific ideas. Towards this goal, we introduce PoT, a semi-verifiable benchmarking framework that links scientific idea judgments to downstream signals that become observable later (e.g., citations and shifts in researchers' agendas). PoT freezes a pre-cutoff snapshot of evidence in an offline sandbox and asks models to forecast post-cutoff outcomes, enabling verifiable evaluation when ground truth arrives, scalable benchmarking without exhaustive expert annotation, and analysis of human-model misalignment against signals such as peer-review awards. In addition, PoT provides a controlled testbed for agent-based research judgments that evaluate scientific ideas, comparing tool-using agents to non-agent baselines under prompt ablations and budget scaling. Across 30,000+ instances spanning four benchmark domains, we find that, compared with non-agent baselines, higher interaction budgets generally improve agent performance, while the benefit of tool use is strongly task-dependent. By combining time-partitioned, future-verifiable targets with an offline sandbox for tool use, PoT supports scalable evaluation of agents on future-facing scientific idea judgment tasks.