Attention as Robust Representation for Time Series Forecasting

TL;DR

This paper proposes elevating attention weights as the main data representation in time series forecasting, demonstrating improved robustness and accuracy over existing models by leveraging attention maps structured with global and local features.

Contribution

It introduces a novel approach that uses attention weights as the primary representation, enhancing robustness against noise and distribution shifts in time series forecasting.

Findings

Outperforms state-of-the-art models with 3.6% lower MSE

Attention maps serve as robust kernels for noisy data

Method is compatible with existing transformer architectures

Abstract

Time series forecasting is essential for many practical applications, with the adoption of transformer-based models on the rise due to their impressive performance in NLP and CV. Transformers' key feature, the attention mechanism, dynamically fusing embeddings to enhance data representation, often relegating attention weights to a byproduct role. Yet, time series data, characterized by noise and non-stationarity, poses significant forecasting challenges. Our approach elevates attention weights as the primary representation for time series, capitalizing on the temporal relationships among data points to improve forecasting accuracy. Our study shows that an attention map, structured using global landmarks and local windows, acts as a robust kernel representation for data points, withstanding noise and shifts in distribution. Our method outperforms state-of-the-art models, reducing mean squared error (MSE) in multivariate time series forecasting by a notable 3.6% without altering the core neural network architecture. It serves as a versatile component that can readily replace recent patching based embedding schemes in transformer-based models, boosting their performance.