On free energy barriers in Gaussian priors and failure of cold start MCMC for high-dimensional unimodal distributions

TL;DR

This paper demonstrates that in high-dimensional Bayesian models with Gaussian priors, MCMC algorithms can experience exponential delays in reaching the main posterior regions, especially with cold start initializations.

Contribution

It provides the first examples showing exponential mixing times for MCMC in high-dimensional unimodal posteriors with Gaussian priors, highlighting limitations of local algorithms.

Findings

MCMC can take exponential time to reach high-probability regions in certain high-dimensional models.

Cold start initializations significantly affect the convergence speed of MCMC algorithms.

The results apply broadly to gradient-based and random walk MCMC schemes, including pCN and MALA.

Abstract

We exhibit examples of high-dimensional unimodal posterior distributions arising in non-linear regression models with Gaussian process priors for which MCMC methods can take an exponential run-time to enter the regions where the bulk of the posterior measure concentrates. Our results apply to worst-case initialised (`cold start') algorithms that are local in the sense that their step-sizes cannot be too large on average. The counter-examples hold for general MCMC schemes based on gradient or random walk steps, and the theory is illustrated for Metropolis-Hastings adjusted methods such as pCN and MALA.