Introducing Spectral Attention for Long-Range Dependency in Time Series Forecasting

TL;DR

This paper introduces Spectral Attention, a novel mechanism that enhances long-range dependency modeling in time series forecasting by preserving temporal correlations and enabling models to capture dependencies over thousands of steps.

Contribution

The paper presents Spectral Attention, a new attention mechanism that can be integrated into existing models to improve long-range dependency capture in time series forecasting.

Findings

Achieves state-of-the-art results on 11 real-world datasets.

Effectively captures long-range dependencies over thousands of steps.

Improves performance of various forecasting models with Spectral Attention.

Abstract

Sequence modeling faces challenges in capturing long-range dependencies across diverse tasks. Recent linear and transformer-based forecasters have shown superior performance in time series forecasting. However, they are constrained by their inherent inability to effectively address long-range dependencies in time series data, primarily due to using fixed-size inputs for prediction. Furthermore, they typically sacrifice essential temporal correlation among consecutive training samples by shuffling them into mini-batches. To overcome these limitations, we introduce a fast and effective Spectral Attention mechanism, which preserves temporal correlations among samples and facilitates the handling of long-range information while maintaining the base model structure. Spectral Attention preserves long-period trends through a low-pass filter and facilitates gradient to flow between samples. Spectral Attention can be seamlessly integrated into most sequence models, allowing models with fixed-sized look-back windows to capture long-range dependencies over thousands of steps. Through extensive experiments on 11 real-world time series datasets using 7 recent forecasting models, we consistently demonstrate the efficacy of our Spectral Attention mechanism, achieving state-of-the-art results.