Sampling hyperparameters in hierarchical models: improving on Gibbs for high-dimensional latent fields and large data sets

TL;DR

This paper introduces a low-dimensional MCMC approach for Bayesian hierarchical models that improves computational efficiency by focusing on hyperparameters, especially in large data settings, without requiring conjugate priors.

Contribution

The authors propose a novel MCMC method that samples hyperparameters directly, bypassing high-dimensional latent variables, and works with non-conjugate hyperpriors, enhancing scalability and flexibility.

Findings

Significant speed-ups in sampling hyperparameters in large datasets.

Method works with non-conjugate hyperpriors.

Efficiency demonstrated in four computational examples.

Abstract

We consider posterior sampling in the very common Bayesian hierarchical model in which observed data depends on high-dimensional latent variables that, in turn, depend on relatively few hyperparameters. When the full conditional over the latent variables has a known form, the marginal posterior distribution over hyperparameters is accessible and can be sampled using a Markov chain Monte Carlo (MCMC) method on a low-dimensional parameter space. This may improve computational efficiency over standard Gibbs sampling since computation is not over the high-dimensional space of latent variables and correlations between hyperparameters and latent variables become irrelevant. When the marginal posterior over hyperparameters depends on a fixed-dimensional sufficient statistic, precomputation of the sufficient statistic renders the cost of the low-dimensional MCMC independent of data size. Then, when the hyperparameters are the primary variables of interest, inference may be performed in big-data settings at modest cost. Moreover, since the form of the full conditional for the latent variables does not depend on the form of the hyperprior distribution, the method imposes no restriction on the hyperprior, unlike Gibbs sampling that typically requires conjugate distributions. We demonstrate these efficiency gains in four computed examples.