On randomized step sizes in Metropolis-Hastings algorithms

TL;DR

This paper investigates the use of randomized step sizes in Metropolis-Hastings algorithms, demonstrating improved robustness, preserved spectral properties, and enhanced acceptance rates, supported by theoretical analysis and numerical experiments.

Contribution

It provides theoretical insights into the spectral properties and robustness of randomized step size algorithms, and compares auxiliary-variable and marginalized constructions.

Findings

Randomized step sizes inherit spectral gaps from fixed-step algorithms.

Marginalized kernels outperform auxiliary-variable ones in asymptotic variance.

Randomization increases robustness and acceptance rates in high-dimensional settings.

Abstract

The performance of Metropolis-Hastings algorithms is highly sensitive to the choice of step size, and miss-specification can lead to severe loss of efficiency. We study algorithms with randomized step sizes, considering both auxiliary-variable and marginalized constructions. We show that algorithms with a randomized step size inherit weak Poincar\'e inequalities/spectral gaps from their fixed-step-size counterparts under minimal conditions, and that the marginalized kernel should always be preferred in terms of asymptotic variance to the auxiliary-variable choice if it is implementable. In addition we show that both types of randomization make an algorithm robust to tuning, meaning that spectral gaps decay polynomially as the step size is increasingly poorly chosen. We further show that step-size randomization often preserves high-dimensional scaling limits and algorithmic complexity, while increasing the optimal acceptance rate for Langevin and Hamiltonian samplers when an Exponential or Uniform distribution is chosen to randomize the step size. Theoretical results are complemented with a numerical study on challenging benchmarks such as Poisson regression, Neal's funnel and the Rosenbrock (banana) distribution.