Dropout Prediction Uncertainty Estimation Using Neuron Activation Strength

TL;DR

This paper proposes a resource-efficient method to estimate dropout prediction uncertainty using neuron activation strengths, eliminating the need for multiple inference runs and maintaining high accuracy across various datasets.

Contribution

The authors introduce a novel approach that uses neuron activation features to estimate dropout uncertainty in a single inference, reducing computational costs significantly.

Findings

Activation strengths can effectively estimate dropout uncertainty.

Using a subset of layers suffices for near-optimal uncertainty estimation.

The method performs well across diverse datasets.

Abstract

Dropout has been commonly used to quantify prediction uncertainty, i.e, the variations of model predictions on a given input example. However, using dropout in practice can be expensive as it requires running dropout inferences many times. In this paper, we study how to estimate dropout prediction uncertainty in a resource-efficient manner. We demonstrate that we can use neuron activation strengths to estimate dropout prediction uncertainty under different dropout settings and on a variety of tasks using three large datasets, MovieLens, Criteo, and EMNIST. Our approach provides an inference-once method to estimate dropout prediction uncertainty as a cheap auxiliary task. We also demonstrate that using activation features from a subset of the neural network layers can be sufficient to achieve uncertainty estimation performance almost comparable to that of using activation features from all layers, thus reducing resources even further for uncertainty estimation.