ImageNet-OOD: Deciphering Modern Out-of-Distribution Detection Algorithms

TL;DR

This paper introduces ImageNet-OOD, a dataset for evaluating out-of-distribution detection algorithms focusing on semantic shift, revealing that current detectors are more sensitive to covariate shift and offering insights for future improvements.

Contribution

The paper presents ImageNet-OOD, a dataset designed to decouple semantic and covariate shifts, and provides a comprehensive analysis of current OOD detection algorithms' sensitivities.

Findings

Current OOD detectors are more sensitive to covariate shift.

Recent algorithms show minimal improvement over simple baselines.

ImageNet-OOD effectively isolates semantic shift for evaluation.

Abstract

The task of out-of-distribution (OOD) detection is notoriously ill-defined. Earlier works focused on new-class detection, aiming to identify label-altering data distribution shifts, also known as "semantic shift." However, recent works argue for a focus on failure detection, expanding the OOD evaluation framework to account for label-preserving data distribution shifts, also known as "covariate shift." Intriguingly, under this new framework, complex OOD detectors that were previously considered state-of-the-art now perform similarly to, or even worse than the simple maximum softmax probability baseline. This raises the question: what are the latest OOD detectors actually detecting? Deciphering the behavior of OOD detection algorithms requires evaluation datasets that decouples semantic shift and covariate shift. To aid our investigations, we present ImageNet-OOD, a clean semantic shift dataset that minimizes the interference of covariate shift. Through comprehensive experiments, we show that OOD detectors are more sensitive to covariate shift than to semantic shift, and the benefits of recent OOD detection algorithms on semantic shift detection is minimal. Our dataset and analyses provide important insights for guiding the design of future OOD detectors.