WAVE: Weighted Autoregressive Varying Gate for Time Series Forecasting

TL;DR

This paper introduces WAVE, an attention mechanism combining autoregressive and moving-average components, which enhances time series forecasting by capturing both long-range and local patterns, achieving state-of-the-art results.

Contribution

The paper presents WAVE, a novel attention mechanism integrating AR and MA components, improving time series forecasting performance and model flexibility.

Findings

WAVE improves forecasting accuracy across multiple datasets.

Incorporating ARMA structure enhances attention models' ability to capture temporal patterns.

WAVE achieves state-of-the-art results in time series forecasting.

Abstract

We propose a Weighted Autoregressive Varying gatE (WAVE) attention mechanism equipped with both Autoregressive (AR) and Moving-average (MA) components. It can adapt to various attention mechanisms, enhancing and decoupling their ability to capture long-range and local temporal patterns in time series data. In this paper, we first demonstrate that, for the time series forecasting (TSF) task, the previously overlooked decoder-only autoregressive Transformer model can achieve results comparable to the best baselines when appropriate tokenization and training methods are applied. Moreover, inspired by the ARMA model from statistics and recent advances in linear attention, we introduce the full ARMA structure into existing autoregressive attention mechanisms. By using an indirect MA weight generation method, we incorporate the MA term while maintaining the time complexity and parameter size of the underlying efficient attention models. We further explore how indirect parameter generation can produce implicit MA weights that align with the modeling requirements for local temporal impacts. Experimental results show that WAVE attention that incorporates the ARMA structure consistently improves the performance of various AR attentions on TSF tasks, achieving state-of-the-art results.