Xihe: Scalable Zero-Shot Time Series Learner Via Hierarchical Interleaved Block Attention

TL;DR

Xihe introduces a hierarchical interleaved block attention mechanism for scalable zero-shot time series modeling, enabling effective multi-scale dependency capture and achieving state-of-the-art results across various model sizes.

Contribution

The paper proposes HIBA, a novel attention architecture for time series models, and develops Xihe, a scalable family of models that excel in zero-shot transfer tasks.

Findings

Xihe-tiny outperforms many existing models with only 9.5M parameters.

Xihe-max (1.5B) achieves new state-of-the-art zero-shot performance.

HIBA effectively captures multi-scale dependencies in time series data.

Abstract

The rapid advancement of time series foundation models (TSFMs) has been propelled by migrating architectures from language models. While existing TSFMs demonstrate impressive performance, their direct adoption of cross-domain architectures constrains effective capture of multiscale temporal dependencies inherent to time series data. This limitation becomes particularly pronounced during zero-shot transfer across datasets with divergent underlying patterns and sampling strategies. To address these challenges, we propose Hierarchical Interleaved Block Attention (HIBA) which employs hierarchical inter- and intra-block sparse attention to effectively capture multi-scale dependencies. Intra-block attention facilitates local information exchange, and inter-block attention operates across blocks to capture global temporal pattern interaction and dynamic evolution. Leveraging the HIBA architecture, we introduce Xihe, a scalable TSFM family spanning from an ultra-efficient 9.5M parameter configuration to high-capacity 1.5B variant. Evaluated on the comprehensive GIFT-Eval benchmark, our most compact Xihe-tiny model (9.5M) surpasses the majority of contemporary TSFMs, demonstrating remarkable parameter efficiency. More impressively, Xihe-max (1.5B) establishes new state-of-the-art zero-shot performance, surpassing previous best results by a substantial margin. This consistent performance excellence across the entire parameter spectrum provides compelling evidence for the exceptional generalization capabilities and architectural superiority of HIBA.