Addressing the Inconsistency in Bayesian Deep Learning via Generalized Laplace Approximation

TL;DR

This paper introduces a generalized Laplace approximation method to improve Bayesian deep learning by addressing model misspecification and uncertainty calibration, leading to better predictive performance.

Contribution

It proposes a novel generalized Laplace approximation that modifies the Hessian calculation, offering a scalable and effective approach for high-quality posterior inference in deep learning.

Findings

Enhanced predictive accuracy on neural networks and real-world datasets

Effective correction for model misspecification and uncertainty calibration

Scalable framework suitable for high-dimensional models

Abstract

In recent years, inconsistency in Bayesian deep learning has attracted significant attention. Tempered or generalized posterior distributions are frequently employed as direct and effective solutions. Nonetheless, the underlying mechanisms and the effectiveness of generalized posteriors remain active research topics. In this work, we interpret posterior tempering as a correction for model misspecification via adjustments to the joint probability, and as a recalibration of priors by reducing aleatoric uncertainty. We also introduce the generalized Laplace approximation, which requires only a simple modification to the Hessian calculation of the regularized loss and provides a flexible and scalable framework for high-quality posterior inference. We evaluate the proposed method on state-of-the-art neural networks and real-world datasets, demonstrating that the generalized Laplace approximation enhances predictive performance.