To MCMC or not to MCMC: Evaluating non-MCMC methods for Bayesian penalized regression

TL;DR

This study evaluates non-MCMC methods like variational inference for Bayesian penalized regression, demonstrating significant speed-ups with comparable predictive accuracy in many scenarios, but with context-dependent performance trade-offs.

Contribution

It provides a comprehensive comparison of MCMC and non-MCMC methods in high-dimensional Bayesian penalized regression, highlighting when computational shortcuts are effective.

Findings

Mean-field variational inference often matches MCMC accuracy.

Variational inference reduces runtime by up to 30x in simulations.

Speed-ups of 100-400x observed in empirical datasets with similar predictive performance.

Abstract

Markov Chain Monte Carlo (MCMC) sampling is computationally expensive, especially for complex models. Alternative methods make simplifying assumptions about the posterior to reduce computational burden, but their impact on predictive performance remains unclear. This paper compares MCMC and non-MCMC methods for high-dimensional penalized regression, examining when computational shortcuts are justified for prediction tasks. We conduct a comprehensive simulation study using high-dimensional tabular data, then validate findings with empirical datasets featuring both continuous and binary outcomes. An in-depth analysis of one dataset provides a step-by-step tutorial implementing various algorithms in R. Our results show that mean-field variational inference consistently performs comparably to MCMC methods. In simulations, mean-field VI exhibited 3-90\% higher MSE across scenarios while reducing runtime by 7-30x compared to Hamiltonian Monte Carlo. Empirical datasets revealed dramatic speed-ups (100-400x) in some cases with similar or superior predictive performance. However, performance varied: some cases showed over 100x MSE increases with only 30x speed-ups, highlighting the context-dependent nature of these trade-offs.