Inferring the Optimal Policy using Markov Chain Monte Carlo

TL;DR

This paper introduces a Markov Chain Monte Carlo-based method for reliably estimating the globally optimal stochastic policy in Markov Decision Processes, addressing high variance issues in existing approaches.

Contribution

It proposes a novel MCMC technique that samples from the posterior distribution conditioned on optimality, ensuring convergence to the global optimum in model-free reinforcement learning.

Findings

Provably converges to the globally optimal policy

Achieves similar variance to policy gradient methods

Applicable to real-world control systems

Abstract

This paper investigates methods for estimating the optimal stochastic control policy for a Markov Decision Process with unknown transition dynamics and an unknown reward function. This form of model-free reinforcement learning comprises many real world systems such as playing video games, simulated control tasks, and real robot locomotion. Existing methods for estimating the optimal stochastic control policy rely on high variance estimates of the policy descent. However, these methods are not guaranteed to find the optimal stochastic policy, and the high variance gradient estimates make convergence unstable. In order to resolve these problems, we propose a technique using Markov Chain Monte Carlo to generate samples from the posterior distribution of the parameters conditioned on being optimal. Our method provably converges to the globally optimal stochastic policy, and empirically similar variance compared to the policy gradient.