Local Evaluation of Time Series Anomaly Detection Algorithms

TL;DR

This paper introduces a new, theoretically grounded, and interpretable extension to classical precision and recall metrics for time series anomaly detection, addressing limitations of existing heuristics and enabling local, fine-grained evaluation.

Contribution

It proposes a robust, parameter-free, and local evaluation metric based on 'affiliation' between ground truth and predictions, improving interpretability and resistance to adversarial strategies.

Findings

The new metrics are theoretically justified and robust against adversarial algorithms.

They enable local, fine-grained evaluation of anomaly detection results.

Normalized metrics quantify improvements over random baselines.

Abstract

In recent years, specific evaluation metrics for time series anomaly detection algorithms have been developed to handle the limitations of the classical precision and recall. However, such metrics are heuristically built as an aggregate of multiple desirable aspects, introduce parameters and wipe out the interpretability of the output. In this article, we first highlight the limitations of the classical precision/recall, as well as the main issues of the recent event-based metrics -- for instance, we show that an adversary algorithm can reach high precision and recall on almost any dataset under weak assumption. To cope with the above problems, we propose a theoretically grounded, robust, parameter-free and interpretable extension to precision/recall metrics, based on the concept of ``affiliation'' between the ground truth and the prediction sets. Our metrics leverage measures of duration between ground truth and predictions, and have thus an intuitive interpretation. By further comparison against random sampling, we obtain a normalized precision/recall, quantifying how much a given set of results is better than a random baseline prediction. By construction, our approach keeps the evaluation local regarding ground truth events, enabling fine-grained visualization and interpretation of algorithmic results. We compare our proposal against various public time series anomaly detection datasets, algorithms and metrics. We further derive theoretical properties of the affiliation metrics that give explicit expectations about their behavior and ensure robustness against adversary strategies.