Riemannian Laplace Approximation with the Fisher Metric

TL;DR

This paper improves the Riemannian Laplace approximation for Bayesian inference by developing variants that are more accurate and unbiased, especially with finite data, through theoretical analysis and practical experiments.

Contribution

It introduces two new variants of the Riemannian Laplace approximation that correct biases and are exact at infinite data, extending the theoretical framework.

Findings

New variants outperform previous methods in accuracy.

Theoretical analysis confirms unbiasedness at infinite data.

Practical experiments demonstrate improved approximation quality.

Abstract

Laplace's method approximates a target density with a Gaussian distribution at its mode. It is computationally efficient and asymptotically exact for Bayesian inference due to the Bernstein-von Mises theorem, but for complex targets and finite-data posteriors it is often too crude an approximation. A recent generalization of the Laplace Approximation transforms the Gaussian approximation according to a chosen Riemannian geometry providing a richer approximation family, while still retaining computational efficiency. However, as shown here, its properties depend heavily on the chosen metric, indeed the metric adopted in previous work results in approximations that are overly narrow as well as being biased even at the limit of infinite data. We correct this shortcoming by developing the approximation family further, deriving two alternative variants that are exact at the limit of infinite data, extending the theoretical analysis of the method, and demonstrating practical improvements in a range of experiments.