SCFormer: Structured Channel-wise Transformer with Cumulative Historical State for Multivariate Time Series Forecasting

TL;DR

SCFormer enhances multivariate time series forecasting by integrating temporal constraints into the Transformer architecture and utilizing cumulative historical data through HiPPO, leading to superior predictive performance.

Contribution

Introduces SCFormer, a novel Transformer model with temporal constraints and HiPPO-based cumulative history handling for improved forecasting accuracy.

Findings

SCFormer outperforms baseline models on multiple datasets.

Temporal constraints improve feature learning in time series.

HiPPO enables effective use of long-term historical information.

Abstract

The Transformer model has shown strong performance in multivariate time series forecasting by leveraging channel-wise self-attention. However, this approach lacks temporal constraints when computing temporal features and does not utilize cumulative historical series effectively.To address these limitations, we propose the Structured Channel-wise Transformer with Cumulative Historical state (SCFormer). SCFormer introduces temporal constraints to all linear transformations, including the query, key, and value matrices, as well as the fully connected layers within the Transformer. Additionally, SCFormer employs High-order Polynomial Projection Operators (HiPPO) to deal with cumulative historical time series, allowing the model to incorporate information beyond the look-back window during prediction. Extensive experiments on multiple real-world datasets demonstrate that SCFormer significantly outperforms mainstream baselines, highlighting its effectiveness in enhancing time series forecasting. The code is publicly available at https://github.com/ShiweiGuo1995/SCFormer