Are Anomaly Scores Telling the Whole Story? A Benchmark for Multilevel Anomaly Detection

TL;DR

This paper introduces Multilevel Anomaly Detection (MAD), a new setting and benchmark for evaluating how well anomaly scores reflect severity, addressing limitations of traditional binary anomaly detection models.

Contribution

The paper proposes MAD, a multilevel AD setting, and MAD-Bench, a benchmark for evaluating severity-aware anomaly detection models, along with a comprehensive performance analysis.

Findings

Models vary in their ability to assign severity-aligned scores

Binary detection performance does not always correlate with severity detection

Robustness of models varies across different severity levels

Abstract

Anomaly detection (AD) is a machine learning task that identifies anomalies by learning patterns from normal training data. In many real-world scenarios, anomalies vary in severity, from minor anomalies with little risk to severe abnormalities requiring immediate attention. However, existing models primarily operate in a binary setting, and the anomaly scores they produce are usually based on the deviation of data points from normal data, which may not accurately reflect practical severity. In this paper, we address this gap by making three key contributions. First, we propose a novel setting, Multilevel AD (MAD), in which the anomaly score represents the severity of anomalies in real-world applications, and we highlight its diverse applications across various domains. Second, we introduce a novel benchmark, MAD-Bench, that evaluates models not only on their ability to detect anomalies, but also on how effectively their anomaly scores reflect severity. This benchmark incorporates multiple types of baselines and real-world applications involving severity. Finally, we conduct a comprehensive performance analysis on MAD-Bench. We evaluate models on their ability to assign severity-aligned scores, investigate the correspondence between their performance on binary and multilevel detection, and study their robustness. This analysis offers key insights into improving AD models for practical severity alignment. The code framework and datasets used for the benchmark will be made publicly available.