Is Gibbs sampling faster than Hamiltonian Monte Carlo on GLMs?

TL;DR

This paper compares Gibbs sampling and Hamiltonian Monte Carlo for Bayesian inference in GLMs, showing Gibbs can be faster in high dimensions due to better scaling, but HMC may still have advantages depending on the condition number.

Contribution

It introduces a dimension-efficient Gibbs sampler for GLMs that outperforms HMC in high-dimensional settings and provides a nuanced comparison of their performance.

Findings

Gibbs sampler's time per sweep scales as O(d) instead of O(d^2).

Gibbs achieves higher effective sample size per unit time in high dimensions.

HMC retains advantages in certain scenarios due to better condition number scaling.

Abstract

The Hamiltonian Monte Carlo (HMC) algorithm is often lauded for its ability to effectively sample from high-dimensional distributions. In this paper we challenge the presumed domination of HMC for the Bayesian analysis of GLMs. By utilizing the structure of the compute graph rather than the graphical model, we show a reduction of the time per sweep of a full-scan Gibbs sampler from $O(d^2)$ to $O(d)$, where $d$ is the number of GLM parameters. A simple change to the implementation of the Gibbs sampler allows us to perform Bayesian inference on high-dimensional GLMs that are practically infeasible with traditional Gibbs sampler implementations. We empirically demonstrate a substantial increase in effective sample size per time when comparing our Gibbs algorithms to state-of-the-art HMC algorithms. While Gibbs is superior in terms of dimension scaling, neither Gibbs nor HMC dominate the other: we provide numerical and theoretical evidence that HMC retains an edge in certain circumstances thanks to its advantageous condition number scaling. Interestingly, for GLMs of fixed data size, we observe that increasing dimensionality can stabilize or even decrease condition number, shedding light on the empirical advantage of our efficient Gibbs sampler.