Intrinsic uncertainties and where to find them

TL;DR

This paper presents a unified framework for uncertainty estimation in machine learning, focusing on marginalising hyperparameters to capture various uncertainties and improve reliability without extensive tuning.

Contribution

It introduces a novel framework that generalizes existing methods by marginalising hyperparameters, enhancing practical uncertainty estimation in standard benchmarks.

Findings

Marginalising hyperparameters effectively captures multiple sources of uncertainty.

Some marginalisation techniques provide reliable uncertainty estimates with minimal tuning.

The framework extends and unifies many existing uncertainty estimation methods.

Abstract

We introduce a framework for uncertainty estimation that both describes and extends many existing methods. We consider typical hyperparameters involved in classical training as random variables and marginalise them out to capture various sources of uncertainty in the parameter space. We investigate which forms and combinations of marginalisation are most useful from a practical point of view on standard benchmarking data sets. Moreover, we discuss how some marginalisations may produce reliable estimates of uncertainty without the need for extensive hyperparameter tuning and/or large-scale ensembling.