Understanding Uncertainty in Bayesian Deep Learning

TL;DR

This paper investigates the limitations of Neural Linear Models in estimating uncertainty, identifies causes for underestimation in data-scarce regions, and proposes a new training method to improve uncertainty estimation and incorporate domain knowledge.

Contribution

It introduces a novel training approach for Neural Linear Models that enhances uncertainty estimation and facilitates domain knowledge integration.

Findings

Traditional NLM training underestimates uncertainty in sparse data regions

Proposed method improves predictive uncertainty estimates

Method allows for domain knowledge incorporation

Abstract

Neural Linear Models (NLM) are deep Bayesian models that produce predictive uncertainty by learning features from the data and then performing Bayesian linear regression over these features. Despite their popularity, few works have focused on formally evaluating the predictive uncertainties of these models. Furthermore, existing works point out the difficulties of encoding domain knowledge in models like NLMs, making them unsuitable for applications where interpretability is required. In this work, we show that traditional training procedures for NLMs can drastically underestimate uncertainty in data-scarce regions. We identify the underlying reasons for this behavior and propose a novel training method that can both capture useful predictive uncertainties as well as allow for incorporation of domain knowledge.