Low Rank Transformer for Multivariate Time Series Anomaly Detection and Localization

TL;DR

This paper introduces ALoRa-T, a low-rank regularized Transformer model for multivariate time series anomaly detection and localization, providing theoretical insights and outperforming existing methods in accuracy.

Contribution

It presents a novel low-rank regularization for Transformers and a new localization method, advancing the understanding and performance of anomaly diagnosis in multivariate time series.

Findings

Significantly outperforms state-of-the-art in detection accuracy

Effectively localizes anomalies to specific variables

Provides theoretical insights linking Transformers to statistical methods

Abstract

Multivariate time series (MTS) anomaly diagnosis, which encompasses both anomaly detection and localization, is critical for the safety and reliability of complex, large-scale real-world systems. The vast majority of existing anomaly diagnosis methods offer limited theoretical insights, especially for anomaly localization, which is a vital but largely unexplored area. The aim of this contribution is to study the learning process of a Transformer when applied to MTS by revealing connections to statistical time series methods. Based on these theoretical insights, we propose the Attention Low-Rank Transformer (ALoRa-T) model, which applies low-rank regularization to self-attention, and we introduce the Attention Low-Rank score, effectively capturing the temporal characteristics of anomalies. Finally, to enable anomaly localization, we propose the ALoRa-Loc method, a novel approach that associates anomalies to specific variables by quantifying interrelationships among time series. Extensive experiments and real data analysis, show that the proposed methodology significantly outperforms state-of-the-art methods in both detection and localization tasks.