A framework for benchmarking uncertainty in deep regression

TL;DR

This paper introduces a flexible framework for benchmarking uncertainty quantification in deep regression models, using a reference Bayesian method to evaluate the reliability and accuracy of various approaches.

Contribution

It presents a novel, adaptable benchmarking framework with a reference solution for assessing uncertainty quantification in deep regression.

Findings

Framework effectively compares deep regression uncertainty methods

Reliability assessed via coverage probabilities

Accuracy evaluated through uncertainty size

Abstract

We propose a framework for the assessment of uncertainty quantification in deep regression. The framework is based on regression problems where the regression function is a linear combination of nonlinear functions. Basically, any level of complexity can be realized through the choice of the nonlinear functions and the dimensionality of their domain. Results of an uncertainty quantification for deep regression are compared against those obtained by a statistical reference method. The reference method utilizes knowledge of the underlying nonlinear functions and is based on a Bayesian linear regression using a reference prior. Reliability of uncertainty quantification is assessed in terms of coverage probabilities, and accuracy through the size of calculated uncertainties. We illustrate the proposed framework by applying it to current approaches for uncertainty quantification in deep regression. The flexibility, together with the availability of a reference solution, makes the framework suitable for defining benchmark sets for uncertainty quantification.