Semantic or Covariate? A Study on the Intractable Case of Out-of-Distribution Detection

TL;DR

This paper clarifies the ambiguous concept of semantic shift in out-of-distribution detection, defines tractable OOD scenarios, and provides theoretical analysis and experiments to improve understanding and evaluation of OOD detection methods.

Contribution

It introduces precise definitions of semantic and covariate spaces, analyzes intractable OOD cases, and proposes a tractable OOD setting for better evaluation of detection methods.

Findings

Current semantic shift definitions are ambiguous.

Theoretical analysis of intractable OOD detection cases.

Validation of definitions through experiments.

Abstract

The primary goal of out-of-distribution (OOD) detection tasks is to identify inputs with semantic shifts, i.e., if samples from novel classes are absent in the in-distribution (ID) dataset used for training, we should reject these OOD samples rather than misclassifying them into existing ID classes. However, we find the current definition of "semantic shift" is ambiguous, which renders certain OOD testing protocols intractable for the post-hoc OOD detection methods based on a classifier trained on the ID dataset. In this paper, we offer a more precise definition of the Semantic Space and the Covariate Space for the ID distribution, allowing us to theoretically analyze which types of OOD distributions make the detection task intractable. To avoid the flaw in the existing OOD settings, we further define the "Tractable OOD" setting which ensures the distinguishability of OOD and ID distributions for the post-hoc OOD detection methods. Finally, we conduct several experiments to demonstrate the necessity of our definitions and validate the correctness of our theorems.