Label Smoothed Embedding Hypothesis for Out-of-Distribution Detection

TL;DR

This paper introduces a novel unsupervised OOD detection method based on $k$-NN density estimates in model embeddings, demonstrating improved performance when models are trained with label smoothing.

Contribution

It proposes the Label Smoothed Embedding Hypothesis and shows that label smoothing enhances $k$-NN based OOD detection both theoretically and empirically.

Findings

Outperforms existing OOD detection baselines.

Theoretical analysis supports improved $k$-NN performance with label smoothing.

Provides new statistical bounds for $k$-NN OOD detection.

Abstract

Detecting out-of-distribution (OOD) examples is critical in many applications. We propose an unsupervised method to detect OOD samples using a $k$-NN density estimate with respect to a classification model's intermediate activations on in-distribution samples. We leverage a recent insight about label smoothing, which we call the \emph{Label Smoothed Embedding Hypothesis}, and show that one of the implications is that the $k$-NN density estimator performs better as an OOD detection method both theoretically and empirically when the model is trained with label smoothing. Finally, we show that our proposal outperforms many OOD baselines and also provide new finite-sample high-probability statistical results for $k$-NN density estimation's ability to detect OOD examples.