Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awareness

TL;DR

This paper introduces SNGP, a single neural network method that enhances uncertainty estimation by improving distance awareness, matching ensemble performance in prediction and out-of-domain detection.

Contribution

The paper formalizes uncertainty estimation as a minimax problem and proposes SNGP, a simple technique combining weight normalization and Gaussian processes to achieve high-quality uncertainty with one model.

Findings

SNGP achieves competitive accuracy and calibration compared to deep ensembles.

SNGP outperforms other single-model uncertainty methods.

SNGP improves out-of-domain detection capabilities.

Abstract

Bayesian neural networks (BNN) and deep ensembles are principled approaches to estimate the predictive uncertainty of a deep learning model. However their practicality in real-time, industrial-scale applications are limited due to their heavy memory and inference cost. This motivates us to study principled approaches to high-quality uncertainty estimation that require only a single deep neural network (DNN). By formalizing the uncertainty quantification as a minimax learning problem, we first identify input distance awareness, i.e., the model's ability to quantify the distance of a testing example from the training data in the input space, as a necessary condition for a DNN to achieve high-quality (i.e., minimax optimal) uncertainty estimation. We then propose Spectral-normalized Neural Gaussian Process (SNGP), a simple method that improves the distance-awareness ability of modern DNNs, by adding a weight normalization step during training and replacing the output layer with a Gaussian process. On a suite of vision and language understanding tasks and on modern architectures (Wide-ResNet and BERT), SNGP is competitive with deep ensembles in prediction, calibration and out-of-domain detection, and outperforms the other single-model approaches.