On Naive Mean-Field Approximation for high-dimensional canonical GLMs

TL;DR

This paper investigates the accuracy of the Naive Mean-Field approximation in high-dimensional generalized linear models with product priors, providing theoretical conditions for its validity and implications for posterior analysis.

Contribution

It offers new theoretical insights into when the NMF approximation is valid for canonical GLMs and proposes algorithms based on the properties of NMF optimizers.

Findings

Sufficient conditions for NMF approximation tightness.

NMF optimizers are product distributions with quadratic tilt.

NMF optimizer determines posterior credible intervals and prediction performance.

Abstract

We study the validity of the Naive Mean Field (NMF) approximation for canonical GLMs with product priors. This setting is challenging due to the non-conjugacy of the likelihood and the prior. Using the theory of non-linear large deviations (Austin 2019, Chatterjee, Dembo 2016, Eldan 2018), we derive sufficient conditions for the tightness of the NMF approximation to the log-normalizing constant of the posterior distribution. As a second contribution, we establish that under minor conditions on the design, any NMF optimizer is a product distribution where each component is a quadratic tilt of the prior. In turn, this suggests novel iterative algorithms for fitting the NMF optimizer to the target posterior. Finally, we establish that if the NMF optimization problem has a "well-separated maximizer", then this optimizer governs the probabilistic properties of the posterior. Specifically, we derive credible intervals with average coverage guarantees, and characterize the prediction performance on an out-of-sample datapoint in terms of this dominant optimizer.