Trace class Markov chains for the Normal-Gamma Bayesian shrinkage model

TL;DR

This paper analyzes the spectral properties of two Gibbs sampling algorithms for the Normal-Gamma Bayesian shrinkage model, demonstrating that the two-block sampler has superior convergence properties due to its trace class Markov operator.

Contribution

It introduces and rigorously compares a two-block Gibbs sampler to the traditional three-block version, proving the former's trace class property and better convergence behavior.

Findings

Two-block sampler's Markov operator is trace class.

Two-block sampler has geometric convergence.

Comparison with sandwich algorithms shows potential improvements.

Abstract

High-dimensional data, where the number of variables exceeds or is comparable to the sample size, is now pervasive in many scientific applications. In recent years, Bayesian shrinkage models have been developed as effective and computationally feasible tools to analyze such data, especially in the context of linear regression. In this paper, we focus on the Normal-Gamma shrinkage model developed by Griffin and Brown. This model subsumes the popular Bayesian lasso model, and a three-block Gibbs sampling algorithm to sample from the resulting intractable posterior distribution has been developed by Griffin and Brown. We consider an alternative two-block Gibbs sampling algorithm and rigorously demonstrate its advantage over the three-block sampler by comparing specific spectral properties. In particular, we show that the Markov operator corresponding to the two-block sampler is trace class (and hence Hilbert-Schmidt), whereas the operator corresponding to the three-block sampler is not even Hilbert-Schmidt. The trace class property for the two-block sampler implies geometric convergence for the associated Markov chain, which justifies the use of Markov chain CLT's to obtain practical error bounds for MCMC based estimates. Additionally, it facilitates theoretical comparisons of the two-block sampler with sandwich algorithms which aim to improve performance by inserting inexpensive extra steps in between the two conditional draws of the two-block sampler.