MCMC-Based Inference in the Era of Big Data: A Fundamental Analysis of the Convergence Complexity of High-Dimensional Chains

TL;DR

This paper analyzes the convergence challenges of MCMC algorithms in high-dimensional settings, revealing limitations of existing methods and proposing a new framework for understanding and ensuring their effective performance as data dimensions grow.

Contribution

The paper introduces a rigorous framework for analyzing high-dimensional MCMC convergence, demonstrating the severity of convergence issues and proposing methods to achieve bounded geometric convergence rates.

Findings

High-dimensional MCMC chains face significant convergence problems.

Existing convergence rate methods have serious limitations in large dimensions.

Proposed diagnostics and constructions improve convergence guarantees in high dimensions.

Abstract

Markov chain Monte Carlo (MCMC) lies at the core of modern Bayesian methodology, much of which would be impossible without it. Thus, the convergence properties of MCMCs have received significant attention, and in particular, proving (geometric) ergodicity is of critical interest. Trust in the ability of MCMCs to sample from modern-day high-dimensional posteriors, however, has been limited by a widespread perception that these chains typically experience serious convergence problems. In this paper, we first demonstrate that contemporary methods for obtaining convergence rates have serious limitations when the dimension grows. We then propose a framework for rigorously establishing the convergence behavior of commonly used high-dimensional MCMCs. In particular, we demonstrate theoretically the precise nature and severity of the convergence problems of popular MCMCs when implemented in high dimensions, including phase transitions in the convergence rates in various $n$ and $p$ regimes, and a universality result across an entire spectrum of models. We also show that convergence problems effectively eliminate the apparent safeguard of geometric ergodicity. We then demonstrate theoretical principles by which MCMCs can be constructed and analyzed to yield bounded geometric convergence rates even as the dimension $p$ grows without bound. Additionally, we propose a diagnostic tool for establishing convergence.