Improving the Linearized Laplace Approximation via Quadratic Approximations

TL;DR

This paper introduces the Quadratic Laplace Approximation (QLA), an enhancement over the Linearized Laplace Approximation (LLA), providing more accurate Bayesian uncertainty estimates for deep neural networks with minimal additional computational cost.

Contribution

The paper proposes QLA, which approximates second-order factors using rank-one updates to improve posterior fidelity without significant computational overhead.

Findings

QLA improves uncertainty estimation over LLA on regression datasets.

QLA maintains computational efficiency similar to LLA.

Empirical results show consistent uncertainty quantification improvements.

Abstract

Deep neural networks (DNNs) often produce overconfident out-of-distribution predictions, motivating Bayesian uncertainty quantification. The Linearized Laplace Approximation (LLA) achieves this by linearizing the DNN and applying Laplace inference to the resulting model. Importantly, the linear model is also used for prediction. We argue this linearization in the posterior may degrade fidelity to the true Laplace approximation. To alleviate this problem, without increasing significantly the computational cost, we propose the Quadratic Laplace Approximation (QLA). QLA approximates each second order factor in the approximate Laplace log-posterior using a rank-one factor obtained via efficient power iterations. QLA is expected to yield a posterior precision closer to that of the full Laplace without forming the full Hessian, which is typically intractable. For prediction, QLA also uses the linearized model. Empirically, QLA yields modest yet consistent uncertainty estimation improvements over LLA on five regression datasets.