When and How Does In-Distribution Label Help Out-of-Distribution Detection?

TL;DR

This paper provides a theoretical framework to understand when and how in-distribution labels improve out-of-distribution detection, using graph spectral analysis to derive error bounds and validate findings empirically.

Contribution

It introduces a formal, graph-theoretic approach to analyze the impact of ID labels on OOD detection, offering conditions for improved performance and empirical validation.

Findings

ID labels can significantly enhance OOD detection under certain conditions.

Spectral decomposition reveals data separability and label influence.

Theoretical bounds are validated through experiments on real and simulated data.

Abstract

Detecting data points deviating from the training distribution is pivotal for ensuring reliable machine learning. Extensive research has been dedicated to the challenge, spanning classical anomaly detection techniques to contemporary out-of-distribution (OOD) detection approaches. While OOD detection commonly relies on supervised learning from a labeled in-distribution (ID) dataset, anomaly detection may treat the entire ID data as a single class and disregard ID labels. This fundamental distinction raises a significant question that has yet to be rigorously explored: when and how does ID label help OOD detection? This paper bridges this gap by offering a formal understanding to theoretically delineate the impact of ID labels on OOD detection. We employ a graph-theoretic approach, rigorously analyzing the separability of ID data from OOD data in a closed-form manner. Key to our approach is the characterization of data representations through spectral decomposition on the graph. Leveraging these representations, we establish a provable error bound that compares the OOD detection performance with and without ID labels, unveiling conditions for achieving enhanced OOD detection. Lastly, we present empirical results on both simulated and real datasets, validating theoretical guarantees and reinforcing our insights. Code is publicly available at https://github.com/deeplearning-wisc/id_label.