Evaluating Strategic Reasoning in Forecasting Agents

TL;DR

This paper introduces BTF-2, a benchmark with 1,417 forecasting questions and reasoning traces, enabling evaluation of strategic reasoning and research-based forecasting accuracy.

Contribution

It presents a new benchmark and a research-based forecaster that outperforms frontier agents, highlighting the importance of pre-mortem analysis and strategic reasoning.

Findings

BTF-2 detects small accuracy differences of 0.004 Brier score.

The research-based forecaster is 0.011 Brier more accurate than any single frontier agent.

Better forecasters analyze their blind spots and consider black swans.

Abstract

Forecasting benchmarks produce accuracy leaderboards but little insight into why some forecasters are more accurate than others. We introduce Bench to the Future 2 (BTF-2), 1,417 pastcasting questions with a frozen 15M-document research corpus in which agents reproducibly research and forecast offline, producing full reasoning traces. BTF-2 detects accuracy differences of 0.004 Brier score, and can distinguish differential agent strengths in research vs. judgment. We build a forecaster 0.011 Brier more accurate than any single frontier agent, and use it to evaluate agent strategic reasoning without hindsight bias. We find the better forecaster differs primarily in its pre-mortem analysis of its blind spots and consideration of black swans. Expert human forecasters found the dominant strategic reasoning failures of frontier agents are in assessing political and business leaders' incentives, judging their likelihood to follow through on stated plans, and modeling institutional processes.