Density Uncertainty Layers for Reliable Uncertainty Estimation

TL;DR

This paper introduces density uncertainty layers, a novel neural network architecture that improves the reliability of predictive uncertainty estimates by grounding uncertainty in the empirical input density, enhancing out-of-distribution detection.

Contribution

The paper proposes a new criterion for reliable uncertainty estimation and develops density uncertainty layers that inherently satisfy this criterion.

Findings

More reliable uncertainty estimates than existing methods

Enhanced out-of-distribution detection performance

Validated on UCI and CIFAR benchmarks

Abstract

Assessing the predictive uncertainty of deep neural networks is crucial for safety-related applications of deep learning. Although Bayesian deep learning offers a principled framework for estimating model uncertainty, the common approaches that approximate the parameter posterior often fail to deliver reliable estimates of predictive uncertainty. In this paper, we propose a novel criterion for reliable predictive uncertainty: a model's predictive variance should be grounded in the empirical density of the input. That is, the model should produce higher uncertainty for inputs that are improbable in the training data and lower uncertainty for inputs that are more probable. To operationalize this criterion, we develop the density uncertainty layer, a stochastic neural network architecture that satisfies the density uncertain criterion by design. We study density uncertainty layers on the UCI and CIFAR-10/100 uncertainty benchmarks. Compared to existing approaches, density uncertainty layers provide more reliable uncertainty estimates and robust out-of-distribution detection performance.