No Free Lunch for Approximate MCMC

TL;DR

This paper demonstrates that subsampling MCMC algorithms, despite their popularity for large datasets, are fundamentally limited in performance improvements due to existing convergence results, with implications for their design.

Contribution

The paper applies theoretical convergence results to subsampling MCMC, revealing inherent limitations and proposing design principles, along with new bounds for random matrix singular values.

Findings

Subsampling MCMC cannot significantly outperform traditional methods due to convergence constraints.

Theoretical bounds explain performance limitations of subsampling algorithms.

Guidelines for designing more effective subsampling MCMC methods.

Abstract

It is widely known that the performance of Markov chain Monte Carlo (MCMC) can degrade quickly when targeting computationally expensive posterior distributions, such as when the sample size is large. This has motivated the search for MCMC variants that scale well to large datasets. One popular general approach has been to look at only a subsample of the data at every step. In this note, we point out that well-known MCMC convergence results often imply that these ``subsampling'' MCMC algorithms cannot greatly improve performance. We apply these abstract results to realistic statistical problems and proposed algorithms, and also discuss some design principles suggested by the results. Finally, we develop estimates for the singular values of random matrices bounds that may be of independent interest.