Laplace Approximation in High-dimensional Bayesian Regression

TL;DR

This paper extends classical results on the Laplace approximation's accuracy to high-dimensional Bayesian regression, providing uniform accuracy across many models and establishing consistency in variable selection.

Contribution

It develops a theory for the uniform accuracy of Laplace approximation in high-dimensional settings and applies it to prove Bayesian variable selection consistency.

Findings

Laplace approximation remains accurate uniformly across models as p and q grow with n.

The results enable consistency proofs for Bayesian variable selection methods.

The theory applies to generalized linear models in high-dimensional regimes.

Abstract

We consider Bayesian variable selection in sparse high-dimensional regression, where the number of covariates $p$ may be large relative to the samples size $n$, but at most a moderate number $q$ of covariates are active. Specifically, we treat generalized linear models. For a single fixed sparse model with well-behaved prior distribution, classical theory proves that the Laplace approximation to the marginal likelihood of the model is accurate for sufficiently large sample size $n$. We extend this theory by giving results on uniform accuracy of the Laplace approximation across all models in a high-dimensional scenario in which $p$ and $q$, and thus also the number of considered models, may increase with $n$. Moreover, we show how this connection between marginal likelihood and Laplace approximation can be used to obtain consistency results for Bayesian approaches to variable selection in high-dimensional regression.