TS-Haystack: A Multi-Task Retrieval Benchmark for Long-Context Time-Series Reasoning

TL;DR

This paper introduces TS-Haystack, a comprehensive benchmark for evaluating long-context reasoning in time-series language models across multiple domains and tasks.

Contribution

It presents TS-Haystack, a new multi-domain retrieval benchmark, and demonstrates that agentic retrieval frameworks outperform existing models on most tasks.

Findings

Existing TSLMs show severe degradation with longer contexts.

Memory limitations cause tokenization models to fail beyond 100 seconds.

Agentic retrieval with specialized tools outperforms state-of-the-art models on most tasks.

Abstract

Time Series Language Models (TSLMs) promise reasoning over real-world temporal data, but their ability to retrieve and reason over long time-series remains largely untested. We introduce TS-Haystack, a multi-domain retrieval benchmark with ten event-grounded question-answering tasks over contexts from 100 seconds to 24 hours, spanning direct retrieval, temporal reasoning, multi-step reasoning, and contextual anomaly detection. Existing TSLMs exhibit severe long-context degradation: accuracy declines with context length, direct-tokenization models run out of memory beyond 100 seconds on high-rate signals, and time-interval-grounded tasks collapse toward near-zero accuracy when increasing the time-series lengths, aligning with existing literature on text and multi-modal long context retrieval. An agentic retrieval framework using specialized time-series classifier tools matches or outperforms SoTA TSLMs on 9 of 10 tasks, highlighting agentic retrieval as a promising approach for long-context TSLMs.