Correlated Attention in Transformers for Multivariate Time Series

TL;DR

This paper introduces a correlated attention mechanism for Transformers that better captures feature cross-correlations and lagged dependencies in multivariate time series, improving performance across various tasks.

Contribution

It proposes a novel correlated attention mechanism that integrates into Transformer encoders to effectively model cross-feature and lagged dependencies in multivariate time series.

Findings

Achieves state-of-the-art results in imputation, anomaly detection, and classification.

Enhances Transformer models with correlated attention for better feature dependency modeling.

Demonstrates consistent improvements across multiple multivariate time series tasks.

Abstract

Multivariate time series (MTS) analysis prevails in real-world applications such as finance, climate science and healthcare. The various self-attention mechanisms, the backbone of the state-of-the-art Transformer-based models, efficiently discover the temporal dependencies, yet cannot well capture the intricate cross-correlation between different features of MTS data, which inherently stems from complex dynamical systems in practice. To this end, we propose a novel correlated attention mechanism, which not only efficiently captures feature-wise dependencies, but can also be seamlessly integrated within the encoder blocks of existing well-known Transformers to gain efficiency improvement. In particular, correlated attention operates across feature channels to compute cross-covariance matrices between queries and keys with different lag values, and selectively aggregate representations at the sub-series level. This architecture facilitates automated discovery and representation learning of not only instantaneous but also lagged cross-correlations, while inherently capturing time series auto-correlation. When combined with prevalent Transformer baselines, correlated attention mechanism constitutes a better alternative for encoder-only architectures, which are suitable for a wide range of tasks including imputation, anomaly detection and classification. Extensive experiments on the aforementioned tasks consistently underscore the advantages of correlated attention mechanism in enhancing base Transformer models, and demonstrate our state-of-the-art results in imputation, anomaly detection and classification.