A variational Bayes approach to debiased inference for low-dimensional parameters in high-dimensional linear regression

TL;DR

This paper introduces a scalable variational Bayes method for accurate inference on low-dimensional parameters within high-dimensional linear regression, combining computational efficiency with reliable uncertainty quantification.

Contribution

It develops a mean-field variational Bayes approach tailored for low-dimensional inference in high-dimensional models, with theoretical guarantees and competitive performance.

Findings

Performs competitively with existing methods

Provides theoretical Bernstein--von Mises guarantees

Ensures accurate uncertainty quantification

Abstract

We propose a scalable variational Bayes method for statistical inference for a single or low-dimensional subset of the coordinates of a high-dimensional parameter in sparse linear regression. Our approach relies on assigning a mean-field approximation to the nuisance coordinates and carefully modelling the conditional distribution of the target given the nuisance. This requires only a preprocessing step and preserves the computational advantages of mean-field variational Bayes, while ensuring accurate and reliable inference for the target parameter, including for uncertainty quantification. We investigate the numerical performance of our algorithm, showing that it performs competitively with existing methods. We further establish accompanying theoretical guarantees for estimation and uncertainty quantification in the form of a Bernstein--von Mises theorem.