TFRBench: A Reasoning Benchmark for Evaluating Forecasting Systems

TL;DR

TFRBench is a novel benchmark for evaluating the reasoning abilities of forecasting systems, emphasizing interpretability and causal analysis over traditional accuracy metrics.

Contribution

It introduces a multi-agent framework and reasoning protocol for assessing forecasting models' understanding of dependencies, trends, and external events.

Findings

Reasoning-based evaluation improves forecasting accuracy (e.g., 40.2% to 56.6%).

Off-the-shelf LLMs struggle with reasoning and domain-specific dynamics.

Benchmark promotes interpretable, reasoning-focused evaluation in time-series forecasting.

Abstract

We introduce TFRBench, the first benchmark designed to evaluate the reasoning capabilities of forecasting systems. Traditionally, time-series forecasting has been evaluated solely on numerical accuracy, treating foundation models as ``black boxes.'' Unlike existing benchmarks, TFRBench provides a protocol for evaluating the reasoning generated by forecasting systems--specifically their analysis of cross-channel dependencies, trends, and external events. To enable this, we propose a systematic multi-agent framework that utilizes an iterative verification loop to synthesize numerically grounded reasoning traces. Spanning ten datasets across five domains, our evaluation confirms that this reasoning is causally effective; useful for evaluation; and prompting LLMs with our generated traces significantly improves forecasting accuracy compared to direct numerical prediction (e.g., avg. $\sim40.2\%\to56.6\%)$, validating the quality of our reasoning. Conversely, benchmarking experiments reveal that off-the-shelf LLMs consistently struggle with both reasoning (lower LLM-as-a-Judge scores) and numerical forecasting, frequently failing to capture domain-specific dynamics. TFRBench thus establishes a new standard for interpretable, reasoning-based evaluation in time-series forecasting. Our benchmark is available at: https://tfrbench.github.io