Excess risk analysis for epistemic uncertainty with application to variational inference

TL;DR

This paper provides a theoretical analysis of epistemic uncertainty in Bayesian deep learning, especially variational inference, linking it to excess risk and proposing a new objective to better control uncertainty.

Contribution

It introduces a novel theoretical framework connecting generalization error and epistemic uncertainty, and proposes a new VI objective based on PAC-Bayesian theory.

Findings

Theoretical relations between generalization error and EU measures are established.

The new VI objective improves EU evaluation in experiments.

Proposed method enhances prediction performance and uncertainty control.

Abstract

Bayesian deep learning plays an important role especially for its ability evaluating epistemic uncertainty (EU). Due to computational complexity issues, approximation methods such as variational inference (VI) have been used in practice to obtain posterior distributions and their generalization abilities have been analyzed extensively, for example, by PAC-Bayesian theory; however, little analysis exists on EU, although many numerical experiments have been conducted on it. In this study, we analyze the EU of supervised learning in approximate Bayesian inference by focusing on its excess risk. First, we theoretically show the novel relations between generalization error and the widely used EU measurements, such as the variance and mutual information of predictive distribution, and derive their convergence behaviors. Next, we clarify how the objective function of VI regularizes the EU. With this analysis, we propose a new objective function for VI that directly controls the prediction performance and the EU based on the PAC-Bayesian theory. Numerical experiments show that our algorithm significantly improves the EU evaluation over the existing VI methods.