TemporalBench: A Benchmark for Evaluating LLM-Based Agents on Contextual and Event-Informed Time Series Tasks

TL;DR

TemporalBench is a comprehensive benchmark designed to evaluate the true temporal reasoning capabilities of models across multiple domains, revealing that high forecasting accuracy does not necessarily imply robust contextual or event-aware understanding.

Contribution

We introduce TemporalBench, a multi-domain benchmark with a four-tier taxonomy to diagnose and analyze temporal reasoning in models beyond simple forecasting accuracy.

Findings

Existing models show fragmented strengths in temporal reasoning.

High numerical forecasting accuracy does not guarantee contextual understanding.

Models exhibit systematic failure modes under complex temporal conditions.

Abstract

It is unclear whether strong forecasting performance reflects genuine temporal understanding or the ability to reason under contextual and event-driven conditions. We introduce TemporalBench, a multi-domain benchmark designed to evaluate temporal reasoning behavior under progressively richer informational settings. TemporalBench adopts a four-tier task taxonomy that examines historical structure interpretation, context-free forecasting, contextual temporal reasoning, and event-conditioned prediction across four real-world domains: retail, healthcare, energy, and physical systems. By controlling access to future targets and contextual information, the benchmark enables a diagnostic analysis of whether models can correctly interpret temporal patterns, align them with external context, and adapt predictions when conditions change. Extensive baseline experiments show that strong numerical forecasting accuracy does not reliably translate into robust contextual or event-aware temporal reasoning; instead, existing agent frameworks exhibit fragmented strengths and systematic failure modes that remain largely hidden under forecasting-only benchmarks. The TemporalBench dataset is publicly available at https://huggingface.co/datasets/Melady/TemporalBench, and we additionally provide a public leaderboard at https://huggingface.co/spaces/Melady/TemporalBench_Leaderboard.