A Variational Information Theoretic Approach to Out-of-Distribution Detection

TL;DR

This paper develops a new information-theoretic framework for creating features that improve out-of-distribution detection in neural networks, leading to better performance and explainability.

Contribution

It introduces a variational method using a novel loss functional combining KL divergence and Information Bottleneck for OOD feature construction.

Findings

The proposed method outperforms existing OOD detection techniques on benchmarks.

The framework predicts new shaping functions that enhance OOD detection.

The approach offers a general, explainable way to design OOD features.

Abstract

We present a theory for the construction of out-of-distribution (OOD) detection features for neural networks. We introduce random features for OOD through a novel information-theoretic loss functional consisting of two terms, the first based on the KL divergence separates resulting in-distribution (ID) and OOD feature distributions and the second term is the Information Bottleneck, which favors compressed features that retain the OOD information. We formulate a variational procedure to optimize the loss and obtain OOD features. Based on assumptions on OOD distributions, one can recover properties of existing OOD features, i.e., shaping functions. Furthermore, we show that our theory can predict a new shaping function that out-performs existing ones on OOD benchmarks. Our theory provides a general framework for constructing a variety of new features with clear explainability.