Global attractors and fast-slow reduction for finite-state actor-critic mean dynamics

TL;DR

This paper analyzes the long-term behavior of finite-state actor-critic algorithms, proving the existence of global attractors and demonstrating how the dynamics can be reduced and approximated under certain conditions.

Contribution

It introduces a rigorous mathematical framework for understanding actor-critic mean dynamics, including attractor existence, Lipschitz properties, and fast-slow reduction techniques.

Findings

Existence of a compact global attractor for the autonomous semiflow.

Lipschitz continuity of the invariant-law map under exponential-mixing.

Convergence of the exact flow to the reduced flow as the parameter tends to zero.

Abstract

When a learning algorithm reshapes the data distribution it trains on, the long-run behavior depends on the joint evolution of the policy, the value estimate, and the data distribution. We study finite-state actor-critic mean dynamics on the enlarged phase space $(\theta,w,\mu)$, where $\theta$ is the actor parameter, $w$ is an auxiliary critic state, and $\mu$ is a state-law variable (the distribution over states induced by the current policy). The state-law coordinate follows the exact controlled-Markov equation $\delta \dot\mu = Q_\theta^*\mu$. Under a softmax actor with box confinement (a smooth proxy for parameter clipping), a uniformly coercive linear critic equation, and a Lipschitz generator family $\theta \mapsto Q_\theta$, we prove that for each $\delta > 0$ the resulting autonomous semiflow possesses a compact global attractor. Under a uniform exponential-mixing assumption, we prove that the invariant-law map $\theta \mapsto \mu_\theta$ is Lipschitz and that the reduced invariant-law system on $(\theta,w)$ is well posed. Under an additional pathwise exponential-stability estimate for the non-autonomous fast state equation, we show that the exact flow tracks the reduced flow on every finite time interval up to the initial layer, and that the exact attractors converge upper semicontinuously to the lifted reduced attractor as $\delta \to 0$. We also give a concrete finite-state reference-state minorization condition implying the pathwise hypothesis. All results are formalized in Lean 4 without custom axioms.