Navigating the Metric Maze: A Taxonomy of Evaluation Metrics for Anomaly Detection in Time Series

TL;DR

This paper provides a comprehensive taxonomy and analysis of evaluation metrics for time series anomaly detection, emphasizing the importance of selecting appropriate metrics based on task-specific requirements.

Contribution

It introduces a detailed taxonomy of 20 evaluation metrics, analyzing their properties and suitability for different anomaly detection scenarios in time series data.

Findings

Certain metrics are better suited for specific anomaly detection tasks.

The choice of evaluation metric significantly impacts the assessment of detection methods.

Careful selection of metrics is crucial for accurate evaluation in diverse scenarios.

Abstract

The field of time series anomaly detection is constantly advancing, with several methods available, making it a challenge to determine the most appropriate method for a specific domain. The evaluation of these methods is facilitated by the use of metrics, which vary widely in their properties. Despite the existence of new evaluation metrics, there is limited agreement on which metrics are best suited for specific scenarios and domain, and the most commonly used metrics have faced criticism in the literature. This paper provides a comprehensive overview of the metrics used for the evaluation of time series anomaly detection methods, and also defines a taxonomy of these based on how they are calculated. By defining a set of properties for evaluation metrics and a set of specific case studies and experiments, twenty metrics are analyzed and discussed in detail, highlighting the unique suitability of each for specific tasks. Through extensive experimentation and analysis, this paper argues that the choice of evaluation metric must be made with care, taking into account the specific requirements of the task at hand.