Know Your Limits: Uncertainty Estimation with ReLU Classifiers Fails at Reliable OOD Detection

TL;DR

This paper demonstrates that common uncertainty estimation methods for ReLU classifiers fail at reliably detecting out-of-distribution data, due to fundamental theoretical limitations related to network representations and activation functions.

Contribution

It provides a theoretical explanation for the failure of uncertainty techniques in OOD detection, highlighting the limitations of ReLU networks and standard uncertainty metrics.

Findings

Uncertainty methods cannot reliably identify OOD samples in ReLU classifiers.

Theoretical proof shows confidence levels are unreliable in unseen feature space regions.

Empirical illustrations on synthetic data support the theoretical results.

Abstract

A crucial requirement for reliable deployment of deep learning models for safety-critical applications is the ability to identify out-of-distribution (OOD) data points, samples which differ from the training data and on which a model might underperform. Previous work has attempted to tackle this problem using uncertainty estimation techniques. However, there is empirical evidence that a large family of these techniques do not detect OOD reliably in classification tasks. This paper gives a theoretical explanation for said experimental findings and illustrates it on synthetic data. We prove that such techniques are not able to reliably identify OOD samples in a classification setting, since their level of confidence is generalized to unseen areas of the feature space. This result stems from the interplay between the representation of ReLU networks as piece-wise affine transformations, the saturating nature of activation functions like softmax, and the most widely-used uncertainty metrics.