S2TX: Cross-Attention Multi-Scale State-Space Transformer for Time Series Forecasting

TL;DR

S2TX introduces a cross-attention multi-scale state-space transformer that effectively integrates long and short-range patterns in multivariate time series forecasting, improving performance and communication between variates.

Contribution

The paper proposes S2TX, a novel model combining cross-attention with state-space transformers to unify multi-scale and multivariate time series modeling.

Findings

Achieves state-of-the-art results on seven benchmark datasets.

Maintains low memory footprint while improving accuracy.

Effectively models long and short-range dependencies with variate interactions.

Abstract

Time series forecasting has recently achieved significant progress with multi-scale models to address the heterogeneity between long and short range patterns. Despite their state-of-the-art performance, we identify two potential areas for improvement. First, the variates of the multivariate time series are processed independently. Moreover, the multi-scale (long and short range) representations are learned separately by two independent models without communication. In light of these concerns, we propose State Space Transformer with cross-attention (S2TX). S2TX employs a cross-attention mechanism to integrate a Mamba model for extracting long-range cross-variate context and a Transformer model with local window attention to capture short-range representations. By cross-attending to the global context, the Transformer model further facilitates variate-level interactions as well as local/global communications. Comprehensive experiments on seven classic long-short range time-series forecasting benchmark datasets demonstrate that S2TX can achieve highly robust SOTA results while maintaining a low memory footprint.