Dropout Injection at Test Time for Post Hoc Uncertainty Quantification in Neural Networks

TL;DR

This paper investigates the use of dropout injection at test time as a post hoc method for uncertainty quantification in neural networks, providing guidelines for its effective application and demonstrating its competitiveness through experiments.

Contribution

It is the first comprehensive comparison of dropout injection versus embedded dropout, offering practical guidelines for effective uncertainty estimation without retraining.

Findings

Dropout injection can serve as a practical post hoc uncertainty measure.

Effectiveness depends on proper scaling of the uncertainty.

Experimental results show competitive performance on regression benchmarks.

Abstract

Among Bayesian methods, Monte-Carlo dropout provides principled tools for evaluating the epistemic uncertainty of neural networks. Its popularity recently led to seminal works that proposed activating the dropout layers only during inference for evaluating uncertainty. This approach, which we call dropout injection, provides clear benefits over its traditional counterpart (which we call embedded dropout) since it allows one to obtain a post hoc uncertainty measure for any existing network previously trained without dropout, avoiding an additional, time-consuming training process. Unfortunately, no previous work compared injected and embedded dropout; therefore, we provide the first thorough investigation, focusing on regression problems. The main contribution of our work is to provide guidelines on the effective use of injected dropout so that it can be a practical alternative to the current use of embedded dropout. In particular, we show that its effectiveness strongly relies on a suitable scaling of the corresponding uncertainty measure, and we discuss the trade-off between negative log-likelihood and calibration error as a function of the scale factor. Experimental results on UCI data sets and crowd counting benchmarks support our claim that dropout injection can effectively behave as a competitive post hoc uncertainty quantification technique.