Depth Uncertainty in Neural Networks

TL;DR

This paper introduces a method for estimating uncertainty in neural networks by probabilistically reasoning over network depth, enabling single-pass predictions and demonstrating robustness and calibration in real-world tasks.

Contribution

It presents a novel approach that models uncertainty through network depth, allowing efficient single-pass inference without multiple forward passes.

Findings

Provides well-calibrated uncertainty estimates

Achieves robustness to dataset shift

Maintains competitive accuracy with less computation

Abstract

Existing methods for estimating uncertainty in deep learning tend to require multiple forward passes, making them unsuitable for applications where computational resources are limited. To solve this, we perform probabilistic reasoning over the depth of neural networks. Different depths correspond to subnetworks which share weights and whose predictions are combined via marginalisation, yielding model uncertainty. By exploiting the sequential structure of feed-forward networks, we are able to both evaluate our training objective and make predictions with a single forward pass. We validate our approach on real-world regression and image classification tasks. Our approach provides uncertainty calibration, robustness to dataset shift, and accuracies competitive with more computationally expensive baselines.