Data Invariants to Understand Unsupervised Out-of-Distribution Detection

TL;DR

This paper introduces a formal characterization of unsupervised out-of-distribution detection based on data invariants, explaining the effectiveness of simple Mahalanobis-based methods and guiding future evaluation practices.

Contribution

It proposes a novel invariant-based framework for understanding U-OOD detection and explains why simple Mahalanobis methods perform well, addressing gaps in current evaluation.

Findings

Most state-of-the-art U-OOD methods do not outperform simple Mahalanobis-based detectors.

The invariants of training data underpin the effectiveness of MahaAD.

The approach offers interpretability and better evaluation insights for U-OOD methods.

Abstract

Unsupervised out-of-distribution (U-OOD) detection has recently attracted much attention due its importance in mission-critical systems and broader applicability over its supervised counterpart. Despite this increase in attention, U-OOD methods suffer from important shortcomings. By performing a large-scale evaluation on different benchmarks and image modalities, we show in this work that most popular state-of-the-art methods are unable to consistently outperform a simple anomaly detector based on pre-trained features and the Mahalanobis distance (MahaAD). A key reason for the inconsistencies of these methods is the lack of a formal description of U-OOD. Motivated by a simple thought experiment, we propose a characterization of U-OOD based on the invariants of the training dataset. We show how this characterization is unknowingly embodied in the top-scoring MahaAD method, thereby explaining its quality. Furthermore, our approach can be used to interpret predictions of U-OOD detectors and provides insights into good practices for evaluating future U-OOD methods.