PeriodNet: Boosting the Potential of Attention Mechanism for Time Series Forecasting

TL;DR

PeriodNet introduces a novel attention-based network structure that significantly improves time series forecasting by capturing local, periodic, and global patterns more effectively than existing models.

Contribution

The paper proposes PeriodNet, a new architecture with period attention, sparse mechanisms, and an iterative grouping for cross-variable modeling, enhancing forecasting accuracy.

Findings

Outperforms six state-of-the-art models on eight datasets.

Achieves 22% relative improvement on long sequence forecasting.

Effectively captures local, periodic, and global dependencies in time series.

Abstract

The attention mechanism has demonstrated remarkable potential in sequence modeling, exemplified by its successful application in natural language processing with models such as Bidirectional Encoder Representations from Transformers (BERT) and Generative Pre-trained Transformer (GPT). Despite these advancements, its utilization in time series forecasting (TSF) has yet to meet expectations. Exploring a better network structure for attention in TSF holds immense significance across various domains. In this paper, we present PeriodNet with a brand new structure to forecast univariate and multivariate time series. PeriodNet incorporates period attention and sparse period attention mechanism for analyzing adjacent periods. It enhances the mining of local characteristics, periodic patterns, and global dependencies. For efficient cross-variable modeling, we introduce an iterative grouping mechanism which can directly reduce the cross-variable redundancy. To fully leverage the extracted features on the encoder side, we redesign the entire architecture of the vanilla Transformer and propose a period diffuser for precise multi-period prediction. Through comprehensive experiments conducted on eight datasets, we demonstrate that PeriodNet outperforms six state-of-the-art models in both univariate and multivariate TSF scenarios in terms of mean square error and mean absolute error. In particular, PeriodNet achieves a relative improvement of 22% when forecasting time series with a length of 720, in comparison to other models based on the conventional encoder-decoder Transformer architecture.