Multi Timescale Stochastic Approximation: Stability and Convergence

TL;DR

This paper establishes comprehensive conditions for the stability and convergence of multi-timescale stochastic approximation algorithms, extending previous results to N-timescale systems and applying them to reinforcement learning methods with momentum, off-policy learning, and constrained optimization.

Contribution

It provides the first unified framework with sufficient conditions for stability and convergence of N-timescale stochastic approximation algorithms, including practical RL algorithms with momentum and primal-dual methods.

Findings

Proved stability and convergence for N-timescale stochastic approximation.

Applied framework to momentum-augmented Gradient TD learning.

Guaranteed convergence of off-policy actor-critic and constrained policy optimization.

Abstract

This paper presents the first sufficient conditions that guarantee the stability and almost sure convergence of multi-timescale stochastic approximation (SA) iterates. It extends the existing results on one-timescale and two-timescale SA iterates to general $N$-timescale stochastic recursions, for any $N \geq 1$, using the ordinary differential equation (ODE) method. As an application, we study SA algorithms augmented with heavy-ball momentum in the context of Gradient Temporal Difference (GTD) learning. The added momentum introduces an auxiliary state evolving on an intermediate timescale, yielding a three-timescale recursion. We show that with appropriate momentum parameters, the scheme fits within our framework and converges almost surely to the same fixed point as baseline GTD. The stability and convergence of all iterates including the momentum state follow from our main results without ad hoc bounds. We then study off-policy actor-critic algorithms with a baseline learner, actor, and critic updated on separate timescales. In contrast to prior work, we eliminate projection steps from the actor update and instead use our framework to guarantee stability and almost sure convergence of all components. Finally, we extend the analysis to constrained policy optimization in the average reward setting, where the actor, critic, and dual variables evolve on three distinct timescales, and we verify that the resulting dynamics satisfy the conditions of our general theorem. These examples show how diverse reinforcement learning algorithms covering momentum acceleration, off-policy learning, and primal-dual methods-fit naturally into the proposed multi-timescale framework.