HEARTS: Benchmarking LLM Reasoning on Health Time Series

TL;DR

HEARTS is a comprehensive benchmark that evaluates large language models' reasoning abilities across diverse health time series data, revealing current limitations and guiding future improvements.

Contribution

The paper introduces HEARTS, a unified benchmark with 16 datasets and 110 tasks for assessing LLM reasoning over health time series, addressing existing gaps.

Findings

LLMs underperform compared to specialized models

Performance is weakly correlated with general reasoning scores

LLMs struggle with multi-step temporal reasoning and scale poorly with complexity

Abstract

The rise of large language models (LLMs) has shifted time series analysis from narrow analytics to general-purpose reasoning. Yet, existing benchmarks cover only a small set of health time series modalities and tasks, failing to reflect the diverse domains and extensive temporal dependencies inherent in real-world physiological modeling. To bridge these gaps, we introduce HEARTS (Health Reasoning over Time Series), a unified benchmark for evaluating hierarchical reasoning capabilities of LLMs over general health time series. HEARTS integrates 16 real-world datasets across 12 health domains and 20 signal modalities, and defines a comprehensive taxonomy of 110 tasks grouped into four core capabilities: Perception, Inference, Generation, and Deduction. Evaluating 14 state-of-the-art LLMs on more than 20K test samples reveals intriguing findings. First, LLMs substantially underperform specialized models, and their performance is only weakly related to general reasoning scores. Moreover, LLMs often rely on simple heuristics and struggle with multi-step temporal reasoning. Finally, performance declines with increasing temporal complexity, with similar failure modes within model families, indicating that scaling alone is insufficient. By making these gaps measurable, HEARTS provides a standardized testbed and living benchmark for developing next-generation LLM agents capable of reasoning over diverse health signals.