Optimal Sample Complexity for Single Time-Scale Actor-Critic with Momentum

TL;DR

This paper proves that a single-timescale actor-critic algorithm with momentum achieves the optimal sample complexity of O(ε^{-2}) for policy optimization in finite MDPs, improving previous results.

Contribution

It introduces a variance reduction technique combining STORM with a sample buffer, achieving optimal sample complexity in actor-critic methods.

Findings

Achieves O(ε^{-2}) sample complexity for actor-critic algorithms.

Uses a novel variance reduction method combining STORM and sample buffer.

Compatible with deep learning architectures and practical implementations.

Abstract

We establish an optimal sample complexity of $O(\epsilon^{-2})$ for obtaining an $\epsilon$-optimal global policy using a single-timescale actor-critic (AC) algorithm in infinite-horizon discounted Markov decision processes (MDPs) with finite state-action spaces, improving upon the prior state of the art of $O(\epsilon^{-3})$. Our approach applies STORM (STOchastic Recursive Momentum) to reduce variance in the critic updates. However, because samples are drawn from a nonstationary occupancy measure induced by the evolving policy, variance reduction via STORM alone is insufficient. To address this challenge, we maintain a buffer of small fraction of recent samples and uniformly sample from it for each critic update. Importantly, these mechanisms are compatible with existing deep learning architectures and require only minor modifications, without compromising practical applicability.