Policy Gradients for Optimal Parallel Tempering MCMC

TL;DR

This paper introduces an adaptive temperature selection algorithm for parallel tempering MCMC using policy gradients, improving sampling efficiency in complex distributions.

Contribution

It presents a novel policy gradient-based method for dynamically adjusting temperatures in parallel tempering, enhancing mixing and reducing autocorrelation.

Findings

Lower integrated autocorrelation times compared to traditional methods

Effective in sampling from multi-modal distributions

Outperforms geometric and uniform temperature schemes

Abstract

Parallel tempering is a meta-algorithm for Markov Chain Monte Carlo that uses multiple chains to sample from tempered versions of the target distribution, enhancing mixing in multi-modal distributions that are challenging for traditional methods. The effectiveness of parallel tempering is heavily influenced by the selection of chain temperatures. Here, we present an adaptive temperature selection algorithm that dynamically adjusts temperatures during sampling using a policy gradient approach. Experiments demonstrate that our method can achieve lower integrated autocorrelation times compared to traditional geometrically spaced temperatures and uniform acceptance rate schemes on benchmark distributions.