On the Convergence of Single-Timescale Actor-Critic

TL;DR

This paper proves that a single-timescale actor-critic algorithm for MDPs converges globally to an optimal policy with improved sample complexity, requiring specific step size decay rates.

Contribution

It introduces a new analytical framework and establishes the first global convergence proof with improved sample complexity for single-timescale AC algorithms.

Findings

Converges to a globally optimal policy with sample complexity O(ε^{-3})

Requires actor and critic step sizes to decay as O(k^{-2/3})

Improves upon previous complexity bounds for actor-critic algorithms

Abstract

We analyze the global convergence of the single-timescale actor-critic (AC) algorithm for the infinite-horizon discounted Markov Decision Processes (MDPs) with finite state spaces. To this end, we introduce an elegant analytical framework for handling complex, coupled recursions inherent in the algorithm. Leveraging this framework, we establish that the algorithm converges to an $\epsilon$-close \textbf{globally optimal} policy with a sample complexity of \( O(\epsilon^{-3}) \). This significantly improves upon the existing complexity of $O(\epsilon^{-2})$ to achieve $\epsilon$-close \textbf{stationary policy}, which is equivalent to the complexity of $O(\epsilon^{-4})$ to achieve $\epsilon$-close \textbf{globally optimal} policy using gradient domination lemma. Furthermore, we demonstrate that to achieve this improvement, the step sizes for both the actor and critic must decay as \( O(k^{-\frac{2}{3}}) \) with iteration $k$, diverging from the conventional \( O(k^{-\frac{1}{2}}) \) rates commonly used in (non)convex optimization.