The Laplace approximation accuracy in high dimensions: a refined analysis and new skew adjustment

TL;DR

This paper provides a refined analysis of the Laplace approximation's accuracy in high-dimensional Bayesian inference, introducing a skew correction that significantly improves approximation quality and establishing bounds relating dimension and sample size.

Contribution

It derives a leading order decomposition of LA error, introduces the first skew correction for LA, and proves new bounds on LA accuracy in high dimensions.

Findings

Skew correction improves LA accuracy by an order of magnitude in high dimensions.

Tighter upper bounds and the first lower bounds on LA accuracy in high dimensions.

LA accuracy requires that the dimension squared is much less than the sample size, i.e., d^2 << n.

Abstract

In Bayesian inference, making deductions about a parameter of interest requires one to sample from or compute an integral against a posterior distribution. A popular method to make these computations cheaper in high-dimensional settings is to replace the posterior with its Laplace approximation (LA), a Gaussian distribution. In this work, we derive a leading order decomposition of the LA error, a powerful technique to analyze the accuracy of the approximation more precisely than was possible before. It allows us to derive the first ever skew correction to the LA which provably improves its accuracy by an order of magnitude in the high-dimensional regime. Our approach also enables us to prove both tighter upper bounds on the standard LA and the first ever lower bounds in high dimensions. In particular, we prove that $d^2\ll n$ is in general necessary for accuracy of the LA, where $d$ is dimension and $n$ is sample size. Finally, we apply our theory in two example models: a Dirichlet posterior arising from a multinomial observation, and logistic regression with Gaussian design. In the latter setting, we prove high probability bounds on the accuracy of the LA and skew-corrected LA in powers of $d/\sqrt n$ alone.