Finite-time analysis of single-timescale actor-critic

TL;DR

This paper provides the first finite-time convergence analysis of online single-timescale actor-critic algorithms in continuous state spaces with linear function approximation, showing they find approximate stationary points efficiently.

Contribution

It introduces a novel framework for analyzing error propagation in single-timescale actor-critic methods, establishing convergence guarantees under practical Markovian sampling.

Findings

Achieves $ ilde{O}(rac{1}{\e^2})$ sample complexity for convergence.

Improves to $O(rac{1}{\e^2})$ under i.i.d. sampling.

Provides systematic error control between actor and critic.

Abstract

Actor-critic methods have achieved significant success in many challenging applications. However, its finite-time convergence is still poorly understood in the most practical single-timescale form. Existing works on analyzing single-timescale actor-critic have been limited to i.i.d. sampling or tabular setting for simplicity. We investigate the more practical online single-timescale actor-critic algorithm on continuous state space, where the critic assumes linear function approximation and updates with a single Markovian sample per actor step. Previous analysis has been unable to establish the convergence for such a challenging scenario. We demonstrate that the online single-timescale actor-critic method provably finds an $\epsilon$-approximate stationary point with $\widetilde{\mathcal{O}}(\epsilon^{-2})$ sample complexity under standard assumptions, which can be further improved to $\mathcal{O}(\epsilon^{-2})$ under the i.i.d. sampling. Our novel framework systematically evaluates and controls the error propagation between the actor and critic. It offers a promising approach for analyzing other single-timescale reinforcement learning algorithms as well.