sTransformer: A Modular Approach for Extracting Inter-Sequential and Temporal Information for Time-Series Forecasting

TL;DR

sTransformer is a modular Transformer-based model that effectively captures inter-sequential and temporal information in long-term time-series forecasting, outperforming existing models and serving as a strong baseline.

Contribution

The paper introduces sTransformer, combining Sequence and Temporal Convolutional Networks with a Sequence-guided Mask Attention mechanism for scalable, comprehensive time-series feature extraction.

Findings

Achieves state-of-the-art results on long-term time-series forecasting.

Demonstrates strong performance on various time-series tasks.

Validates effectiveness of Transformer-based structures for time-series analysis.

Abstract

In recent years, numerous Transformer-based models have been applied to long-term time-series forecasting (LTSF) tasks. However, recent studies with linear models have questioned their effectiveness, demonstrating that simple linear layers can outperform sophisticated Transformer-based models. In this work, we review and categorize existing Transformer-based models into two main types: (1) modifications to the model structure and (2) modifications to the input data. The former offers scalability but falls short in capturing inter-sequential information, while the latter preprocesses time-series data but is challenging to use as a scalable module. We propose $\textbf{sTransformer}$, which introduces the Sequence and Temporal Convolutional Network (STCN) to fully capture both sequential and temporal information. Additionally, we introduce a Sequence-guided Mask Attention mechanism to capture global feature information. Our approach ensures the capture of inter-sequential information while maintaining module scalability. We compare our model with linear models and existing forecasting models on long-term time-series forecasting, achieving new state-of-the-art results. We also conducted experiments on other time-series tasks, achieving strong performance. These demonstrate that Transformer-based structures remain effective and our model can serve as a viable baseline for time-series tasks.