Gradient Temporal Difference with Momentum: Stability and Convergence

TL;DR

This paper introduces a momentum-enhanced Gradient TD algorithm for reinforcement learning, providing theoretical convergence guarantees and demonstrating improved performance over traditional methods through empirical evaluation.

Contribution

It extends Gradient TD algorithms with heavy ball momentum, offering the first stability and convergence analysis for three-timescale stochastic approximation in this context.

Findings

Proves convergence of momentum-based Gradient TD algorithms.

Provides stability conditions for three-timescale stochastic approximation.

Shows empirical performance improvements over vanilla algorithms.

Abstract

Gradient temporal difference (Gradient TD) algorithms are a popular class of stochastic approximation (SA) algorithms used for policy evaluation in reinforcement learning. Here, we consider Gradient TD algorithms with an additional heavy ball momentum term and provide choice of step size and momentum parameter that ensures almost sure convergence of these algorithms asymptotically. In doing so, we decompose the heavy ball Gradient TD iterates into three separate iterates with different step sizes. We first analyze these iterates under one-timescale SA setting using results from current literature. However, the one-timescale case is restrictive and a more general analysis can be provided by looking at a three-timescale decomposition of the iterates. In the process, we provide the first conditions for stability and convergence of general three-timescale SA. We then prove that the heavy ball Gradient TD algorithm is convergent using our three-timescale SA analysis. Finally, we evaluate these algorithms on standard RL problems and report improvement in performance over the vanilla algorithms.