Policy Guided Monte Carlo: Reinforcement Learning Markov Chain Dynamics

TL;DR

This paper presents Policy Guided Monte Carlo (PGMC), a reinforcement learning-based framework that autonomously discovers efficient Markov chain Monte Carlo sampling strategies, demonstrated on complex Ising models including kagome spin ice.

Contribution

The paper introduces PGMC, a novel, general, and unbiased reinforcement learning approach for automating the discovery of efficient MCMC samplers, applicable across various models.

Findings

PGMC successfully learns efficient MCMC updates for the kagome lattice Ising model.

PGMC operates without prior physics knowledge, demonstrating automation.

Shows potential for broad applicability in complex statistical models.

Abstract

We introduce \textit{Policy Guided Monte Carlo} (PGMC), a computational framework using reinforcement learning to improve Markov chain Monte Carlo (MCMC) sampling. The methodology is generally applicable, unbiased and opens up a new path to automated discovery of efficient MCMC samplers. After developing a general theory, we demonstrate some of PGMC's prospects on an Ising model on the kagome lattice, including when the model is in its computationally challenging kagome spin ice regime. Here, we show that PGMC is able to automatically machine learn efficient MCMC updates without a priori knowledge of the physics at hand.