TwinFormer: A Dual-Level Transformer for Long-Sequence Time-Series Forecasting

TL;DR

TwinFormer introduces a hierarchical Transformer architecture with local and global attention mechanisms for efficient long-sequence time-series forecasting, achieving state-of-the-art results across diverse datasets.

Contribution

The paper proposes a novel dual-level Transformer architecture that combines local sparse attention and global inter-patch modeling for improved long-term forecasting.

Findings

Outperforms existing models on 8 real-world datasets

Achieves linear time and memory complexity

Demonstrates superior accuracy in MAE and RMSE metrics

Abstract

TwinFormer is a hierarchical Transformer for long-sequence time-series forecasting. It divides the input into non-overlapping temporal patches and processes them in two stages: (1) a Local Informer with top-$k$ Sparse Attention models intra-patch dynamics, followed by mean pooling; (2) a Global Informer captures long-range inter-patch dependencies using the same top-$k$ attention. A lightweight GRU aggregates the globally contextualized patch tokens for direct multi-horizon prediction. The resulting architecture achieves linear $O(kLd)$ time and memory complexity. On eight real-world benchmarking datasets from six different domains, including weather, stock price, temperature, power consumption, electricity, and disease, and forecasting horizons $96-720$, TwinFormer secures $27$ positions in the top two out of $34$. Out of the $27$, it achieves the best performance on MAE and RMSE at $17$ places and $10$ at the second-best place on MAE and RMSE. This consistently outperforms PatchTST, iTransformer, FEDformer, Informer, and vanilla Transformers. Ablations confirm the superiority of top-$k$ Sparse Attention over ProbSparse and the effectiveness of GRU-based aggregation. Code is available at this repository: https://github.com/Mahimakumavat1205/TwinFormer.