Spectral Collapsed Gibbs Sampler for Bayesian Sparse Regression

TL;DR

This paper introduces an efficient spectral collapsed Gibbs sampler for Bayesian sparse regression, improving mixing and convergence by directly sampling the global scale parameter without Metropolis steps.

Contribution

It develops a novel spectral decomposition technique enabling direct sampling of the collapsed density, eliminating the need for Metropolis proposals in Bayesian sparse regression.

Findings

Faster convergence and improved mixing in high-dimensional Bayesian regression.

Successful application to large datasets with hundreds of thousands of features.

Efficient sampling of the global scale parameter without Metropolis steps.

Abstract

Sparse regression based on global-local shrinkage priors are increasingly used for Bayesian modeling of modern high-dimensional data, but scaling up the Gibbs sampler for posterior inference remains a challenge. While much effort has gone into speeding up the high-dimensional coefficient update step, insufficient attention has been given to the potential poor mixing of the global scale parameter $\tau$ and of the overall sampler. One proposed remedy has been to marginalize out the coefficients when updating $\tau$. Here we show that, while this collapsed update was previously thought to require a Metropolis step, we can in fact sample directly and efficiently from the collapsed density. This is made possible by careful linear algebraic manipulations and a strategic per-Gibbs-scan spectral decomposition, allowing subsequent evaluations of the collapsed density across hundreds of values of $\tau$ at negligible cost. We combine this computational trick with adaptive numerical integration and inverse transform sampling to construct a direct sampler. This eliminates the need to tune Metropolis proposals and yields faster convergence and improved mixing. We demonstrate our method on two big data applications, fitting logistic regression under the horseshoe prior to datasets with design matrices of size 120,000 x 1,379 and 1,980 x 17,848.