How to pick the best anomaly detector?

TL;DR

This paper introduces the ARGOS metric, a data-driven and theoretically grounded method for selecting the most sensitive anomaly detector tailored to specific data, outperforming traditional metrics like binary cross-entropy.

Contribution

The paper presents the ARGOS metric, a novel, robust, and theoretically supported approach for model selection in anomaly detection, especially for weakly-supervised, classifier-based methods.

Findings

ARGOS outperforms binary cross-entropy in model selection.

ARGOS is robust to noisy conditions in anomaly detection.

Applicable to hyperparameter tuning, architecture, and feature selection.

Abstract

Anomaly detection has the potential to discover new physics in unexplored regions of the data. However, choosing the best anomaly detector for a given data set in a model-agnostic way is an important challenge which has hitherto largely been neglected. In this paper, we introduce the data-driven ARGOS metric, which has a sound theoretical foundation and is empirically shown to robustly select the most sensitive anomaly detection model given the data. Focusing on weakly-supervised, classifier-based anomaly detection methods, we show that the ARGOS metric outperforms other model selection metrics previously used in the literature, in particular the binary cross-entropy loss. We explore several realistic applications, including hyperparameter tuning as well as architecture and feature selection, and in all cases we demonstrate that ARGOS is robust to the noisy conditions of anomaly detection.