A Two-Timescale Primal-Dual Framework for Reinforcement Learning via Online Dual Variable Guidance

TL;DR

This paper introduces PGDA-RL, a primal-dual algorithm for reinforcement learning that leverages two-timescale stochastic approximation, enabling online policy updates with theoretical convergence guarantees under weaker assumptions.

Contribution

The paper proposes PGDA-RL, a novel two-timescale primal-dual RL algorithm that operates asynchronously with online updates, removing the need for a simulator or fixed policy, and provides convergence analysis.

Findings

PGDA-RL converges almost surely to the optimal policy and value function.

Under ergodicity, PGDA-RL achieves a $ ilde{O}(k^{-2/3})$ convergence rate.

The algorithm works with correlated data and off-policy information.

Abstract

We study reinforcement learning by combining recent advances in regularized linear programming formulations with the classical theory of stochastic approximation. Motivated by the challenge of designing algorithms that leverage off-policy data while maintaining on-policy exploration, we propose PGDA-RL, a novel primal-dual Projected Gradient Descent-Ascent algorithm for solving regularized Markov Decision Processes (MDPs). PGDA-RL integrates experience replay-based gradient estimation with a two-timescale decomposition of the underlying nested optimization problem. The algorithm operates asynchronously, interacts with the environment through a single trajectory of correlated data, and updates its policy online in response to the dual variable associated with the occupancy measure of the underlying MDP. We prove that PGDA-RL converges almost surely to the optimal value function and policy of the regularized MDP. Our convergence analysis relies on tools from stochastic approximation theory and holds under weaker assumptions than those required by existing primal-dual RL approaches, notably removing the need for a simulator or a fixed behavioral policy. Under a strengthened ergodicity assumption on the underlying Markov chain, we establish a last-iterate finite-time guarantee with $\tilde{O} (k^{-2/3})$ mean-square convergence, aligning with the best-known rates for two-timescale stochastic approximation methods under Markovian sampling and biased gradient estimates.