Is AUC the best measure for practical comparison of anomaly detectors?

TL;DR

This paper critically examines the effectiveness of AUC as a metric for anomaly detection, highlighting its limitations and proposing alternative evaluation approaches aligned with practical needs.

Contribution

The study questions AUC's suitability for anomaly detection and suggests that low false positive rate metrics and representative anomalous samples are crucial for meaningful comparisons.

Findings

AUC may not reflect practical detection needs.

Metrics emphasizing low false positive rates are more relevant.

Representative anomalous samples are essential for comparison.

Abstract

The area under receiver operating characteristics (AUC) is the standard measure for comparison of anomaly detectors. Its advantage is in providing a scalar number that allows a natural ordering and is independent on a threshold, which allows to postpone the choice. In this work, we question whether AUC is a good metric for anomaly detection, or if it gives a false sense of comfort, due to relying on assumptions which are unlikely to hold in practice. Our investigation shows that variations of AUC emphasizing accuracy at low false positive rate seem to be better correlated with the needs of practitioners, but also that we can compare anomaly detectors only in the case when we have representative examples of anomalous samples. This last result is disturbing, as it suggests that in many cases, we should do active or few-show learning instead of pure anomaly detection.