On Pixel-level Performance Assessment in Anomaly Detection

TL;DR

This paper investigates the challenges of evaluating pixel-level anomaly detection methods, highlighting the limitations of existing metrics and proposing the use of Precision-Recall-based metrics for more accurate performance assessment.

Contribution

It provides a detailed analysis of evaluation metric limitations and demonstrates that Precision-Recall metrics better reflect true method performance in pixel-level anomaly detection.

Findings

Precision-Recall metrics outperform traditional metrics in capturing method performance

Visual and statistical analysis reveal the inadequacy of existing evaluation metrics

Extensive experiments on 21 problems with 11 methods support the proposed insights

Abstract

Anomaly detection methods have demonstrated remarkable success across various applications. However, assessing their performance, particularly at the pixel-level, presents a complex challenge due to the severe imbalance that is most commonly present between normal and abnormal samples. Commonly adopted evaluation metrics designed for pixel-level detection may not effectively capture the nuanced performance variations arising from this class imbalance. In this paper, we dissect the intricacies of this challenge, underscored by visual evidence and statistical analysis, leading to delve into the need for evaluation metrics that account for the imbalance. We offer insights into more accurate metrics, using eleven leading contemporary anomaly detection methods on twenty-one anomaly detection problems. Overall, from this extensive experimental evaluation, we can conclude that Precision-Recall-based metrics can better capture relative method performance, making them more suitable for the task.