Is Out-of-Distribution Detection Learnable?

TL;DR

This paper explores the theoretical limits of out-of-distribution detection, establishing conditions for its learnability and providing insights into when effective OOD detection is possible or impossible.

Contribution

It introduces a PAC learning framework for OOD detection, proving impossibility results and identifying practical scenarios where OOD detection can be learned.

Findings

Necessary condition for OOD detection learnability

Impossibility theorems under certain scenarios

Conditions for practical learnability of OOD detection

Abstract

Supervised learning aims to train a classifier under the assumption that training and test data are from the same distribution. To ease the above assumption, researchers have studied a more realistic setting: out-of-distribution (OOD) detection, where test data may come from classes that are unknown during training (i.e., OOD data). Due to the unavailability and diversity of OOD data, good generalization ability is crucial for effective OOD detection algorithms. To study the generalization of OOD detection, in this paper, we investigate the probably approximately correct (PAC) learning theory of OOD detection, which is proposed by researchers as an open problem. First, we find a necessary condition for the learnability of OOD detection. Then, using this condition, we prove several impossibility theorems for the learnability of OOD detection under some scenarios. Although the impossibility theorems are frustrating, we find that some conditions of these impossibility theorems may not hold in some practical scenarios. Based on this observation, we next give several necessary and sufficient conditions to characterize the learnability of OOD detection in some practical scenarios. Lastly, we also offer theoretical supports for several representative OOD detection works based on our OOD theory.