Sampling the Bayesian Elastic Net

TL;DR

This paper develops new MCMC algorithms for full Bayesian elastic net regression that avoid Metropolis steps, enabling more efficient and direct sampling of all parameters, and compares these methods empirically.

Contribution

It introduces a novel combination of prior form and representation for the Bayesian elastic net, along with rejection sampling-based MCMC algorithms that eliminate the need for tuning proposal distributions.

Findings

New algorithms enable direct sampling without Metropolis steps

Empirical results show improved efficiency over existing samplers

Comparison across data structures demonstrates robustness and accuracy

Abstract

The Bayesian elastic net regression model is characterized by the regression coefficient prior distribution, the negative log density of which corresponds to the elastic net penalty function. While Markov chain Monte Carlo (MCMC) methods exist for sampling from the posterior of the regression coefficients given the penalty parameters, full Bayesian inference that incorporates uncertainty about the penalty parameters remains a challenge due to an intractable integrable in the posterior density function. Though sampling methods have been proposed that avoid computing this integral, all correctly-specified methods for full Bayesian inference that have appeared in the literature involve at least one "Metropolis-within-Gibbs" update, requiring tuning of proposal distributions. The computational landscape is complicated by the fact that two forms of the Bayesian elastic net prior have been introduced, and two representations (with and without data augmentation) of the prior suggest different MCMC algorithms. We review the forms and representations of the prior, discuss all combinations of these different treatments for the first time, and introduce one combination of form and representation that has yet to appear in the literature. We introduce MCMC algorithms for full Bayesian inference for all treatments of the prior. The algorithms allow for direct sampling of all parameters without any "Metropolis-within-Gibbs" steps. The key to the new approach is a careful transformation of the parameter space and an analysis of the resulting full conditional density functions that allows for efficient rejection sampling. We make empirical comparisons between our approaches and existing MCMC samplers for different data structures.