Multivariate Time Series Anomaly Detection: Fancy Algorithms and Flawed Evaluation Methodology

TL;DR

This paper critically reviews multivariate time series anomaly detection methods, highlighting flawed evaluation protocols, proposing robust alternatives, and demonstrating a simple PCA baseline that outperforms complex deep learning models on benchmarks.

Contribution

It identifies flaws in current evaluation methods, proposes more robust protocols, and introduces a simple PCA baseline that challenges the effectiveness of recent deep learning approaches.

Findings

Flawed point-adjust protocol can be outperformed by random guessing.

A simple PCA baseline outperforms many deep learning models on benchmarks.

Robust evaluation protocols are essential for fair comparison of anomaly detection methods.

Abstract

Multivariate Time Series (MVTS) anomaly detection is a long-standing and challenging research topic that has attracted tremendous research effort from both industry and academia recently. However, a careful study of the literature makes us realize that 1) the community is active but not as organized as other sibling machine learning communities such as Computer Vision (CV) and Natural Language Processing (NLP), and 2) most proposed solutions are evaluated using either inappropriate or highly flawed protocols, with an apparent lack of scientific foundation. So flawed is one very popular protocol, the so-called point-adjust protocol, that a random guess can be shown to systematically outperform all algorithms developed so far. In this paper, we review and evaluate many recent algorithms using more robust protocols and discuss how a normally good protocol may have weaknesses in the context of MVTS anomaly detection and how to mitigate them. We also share our concerns about benchmark datasets, experiment design and evaluation methodology we observe in many works. Furthermore, we propose a simple, yet challenging, baseline based on Principal Components Analysis (PCA) that surprisingly outperforms many recent Deep Learning (DL) based approaches on popular benchmark datasets. The main objective of this work is to stimulate more effort towards important aspects of the research such as data, experiment design, evaluation methodology and result interpretability, instead of putting the highest weight on the design of increasingly more complex and "fancier" algorithms.