Sequence Complementor: Complementing Transformers For Time Series Forecasting with Learnable Sequences

TL;DR

This paper introduces learnable Sequence Complementors to enhance transformer-based time series forecasting by improving sequence representation diversity, leading to better performance in both long-term and short-term predictions.

Contribution

It proposes a novel attention mechanism with learnable Sequence Complementors and a diversification loss, improving transformer performance through more diverse sequence representations.

Findings

Sequence Complementors improve forecasting accuracy.

Enhanced diversity of sequence representations correlates with lower error.

Method outperforms recent state-of-the-art models in experiments.

Abstract

Since its introduction, the transformer has shifted the development trajectory away from traditional models (e.g., RNN, MLP) in time series forecasting, which is attributed to its ability to capture global dependencies within temporal tokens. Follow-up studies have largely involved altering the tokenization and self-attention modules to better adapt Transformers for addressing special challenges like non-stationarity, channel-wise dependency, and variable correlation in time series. However, we found that the expressive capability of sequence representation is a key factor influencing Transformer performance in time forecasting after investigating several representative methods, where there is an almost linear relationship between sequence representation entropy and mean square error, with more diverse representations performing better. In this paper, we propose a novel attention mechanism with Sequence Complementors and prove feasible from an information theory perspective, where these learnable sequences are able to provide complementary information beyond current input to feed attention. We further enhance the Sequence Complementors via a diversification loss that is theoretically covered. The empirical evaluation of both long-term and short-term forecasting has confirmed its superiority over the recent state-of-the-art methods.