Dynamic mean field programming

TL;DR

This paper develops a dynamic mean field theory for large-scale Bayesian reinforcement learning, providing exact and approximate equations to analyze the statistical structure of value functions in complex environments.

Contribution

It introduces dynamic mean field programming (DMFP), a novel theoretical framework that models the distribution of state-action values in large state spaces, bridging statistical physics and reinforcement learning.

Findings

State-action values are statistically independent in the asymptotic limit.

Exact distribution forms for value functions are derived.

Conditions identified under which RL reduces to independent bandit problems.

Abstract

A dynamic mean field theory is developed for finite state and action Bayesian reinforcement learning in the large state space limit. In an analogy with statistical physics, the Bellman equation is studied as a disordered dynamical system; the Markov decision process transition probabilities are interpreted as couplings and the value functions as deterministic spins that evolve dynamically. Thus, the mean-rewards and transition probabilities are considered to be quenched random variables. The theory reveals that, under certain assumptions, the state-action values are statistically independent across state-action pairs in the asymptotic state space limit, and provides the form of the distribution exactly. The results hold in the finite and discounted infinite horizon settings, for both value iteration and policy evaluation. The state-action value statistics can be computed from a set of mean field equations, which we call dynamic mean field programming (DMFP). For policy evaluation the equations are exact. For value iteration, approximate equations are obtained by appealing to extreme value theory or bounds. The result provides analytic insight into the statistical structure of tabular reinforcement learning, for example revealing the conditions under which reinforcement learning is equivalent to a set of independent multi-armed bandit problems.