TimeSeriesExamAgent: Creating Time Series Reasoning Benchmarks at Scale

TL;DR

This paper introduces scalable methods for creating comprehensive time series reasoning benchmarks using synthetic and real-world data, revealing current LLM limitations in understanding and domain-specific applications.

Contribution

It presents TimeSeriesExam and TimeSeriesExamAgent, novel approaches for generating diverse, large-scale time series reasoning benchmarks automatically from real datasets.

Findings

Benchmarks achieve diversity comparable to manual curation

LLMs show limited performance in abstract and domain-specific time series reasoning

Automatic benchmark generation is effective across multiple real-world domains

Abstract

Large Language Models (LLMs) have shown promising performance in time series modeling tasks, but do they truly understand time series data? While multiple benchmarks have been proposed to answer this fundamental question, most are manually curated and focus on narrow domains or specific skill sets. To address this limitation, we propose scalable methods for creating comprehensive time series reasoning benchmarks that combine the flexibility of templates with the creativity of LLM agents. We first develop TimeSeriesExam, a multiple-choice benchmark using synthetic time series to evaluate LLMs across five core reasoning categories: pattern recognitionnoise understandingsimilarity analysisanomaly detection, and causality. Then, with TimeSeriesExamAgent, we scale our approach by automatically generating benchmarks from real-world datasets spanning healthcare, finance and weather domains. Through multi-dimensional quality evaluation, we demonstrate that our automatically generated benchmarks achieve diversity comparable to manually curated alternatives. However, our experiments reveal that LLM performance remains limited in both abstract time series reasoning and domain-specific applications, highlighting ongoing challenges in enabling effective time series understanding in these models. TimeSeriesExamAgent is available at https://github.com/magwiazda/TimeSeriesExamAgent.