Laplace Variational Inference for Bayesian Envelope Models

TL;DR

This paper introduces a novel Laplace variational inference method for Bayesian envelope models, improving computational efficiency and stability over traditional MCMC and ADVI methods, with proven convergence and effective real-data application.

Contribution

It proposes a new reparameterization and a Laplace approximation within a variational inference framework for Bayesian envelope models, addressing ill-conditioning and computational challenges.

Findings

Significantly faster inference compared to MCMC and ADVI.

Maintains estimation accuracy and model selection performance.

Theoretical convergence of the Laplace approximation error.

Abstract

Envelope models provide a sufficient dimension reduction framework for multivariate regression analysis. Bayesian inference for these models has been developed primarily using Markov chain Monte Carlo (MCMC) methods. Specifically, Gibbs sampling and Metropolis-Hastings algorithms suffer from slow mixing and high computational cost. Although automatic differentiation variational inference (ADVI) has been explored for Bayesian envelope models, the resulting gradient-based optimization is often numerically unstable due to severe ill-conditioning of the posterior distribution. To address this issue, we propose a novel reparameterization of the posterior distribution that alleviates the ill-conditioning inherent in conventional variational approaches. Building on this reparameterization, we develop an efficient variational inference procedure. Since the resulting likelihood remains nonconjugate, we approximate the corresponding variational factor using a Laplace approximation within a coordinate-ascent variational inference (CAVI) framework. We establish theoretical results showing that, at each one-step coordinate update, the Laplace approximation error relative to the exact variational inference coordinate update converges to zero. Simulation studies and a real-data analysis demonstrate that the proposed method substantially improves computational efficiency while maintaining estimation accuracy and model-selection performance relative to existing approaches.