Policy Gradient With Serial Markov Chain Reasoning

TL;DR

This paper presents a novel reinforcement learning framework that models decision-making as an iterative reasoning process using a Markov chain, enabling flexible action distributions and adaptive reasoning steps, leading to state-of-the-art results.

Contribution

It introduces a new framework for RL decision-making based on reasoning Markov chains, allowing flexible action distributions and adaptive reasoning, with improved performance.

Findings

Achieves state-of-the-art results on Mujoco and DeepMind Control benchmarks.

Allows approximation of any continuous action distribution.

Enables adaptive scaling of reasoning steps based on decision difficulty.

Abstract

We introduce a new framework that performs decision-making in reinforcement learning (RL) as an iterative reasoning process. We model agent behavior as the steady-state distribution of a parameterized reasoning Markov chain (RMC), optimized with a new tractable estimate of the policy gradient. We perform action selection by simulating the RMC for enough reasoning steps to approach its steady-state distribution. We show our framework has several useful properties that are inherently missing from traditional RL. For instance, it allows agent behavior to approximate any continuous distribution over actions by parameterizing the RMC with a simple Gaussian transition function. Moreover, the number of reasoning steps to reach convergence can scale adaptively with the difficulty of each action selection decision and can be accelerated by re-using past solutions. Our resulting algorithm achieves state-of-the-art performance in popular Mujoco and DeepMind Control benchmarks, both for proprioceptive and pixel-based tasks.