Gradient Iterated Temporal-Difference Learning

TL;DR

This paper introduces Gradient Iterated Temporal-Difference learning, a new gradient-based TD method that improves stability and learning speed, demonstrating competitive performance on benchmarks like Atari games.

Contribution

It develops a novel gradient TD algorithm based on iterated TD learning, addressing stability issues and achieving performance comparable to semi-gradient methods.

Findings

The proposed method is stable and competitive with semi-gradient TD methods.

It demonstrates strong performance on Atari benchmarks.

The algorithm effectively learns sequences of value functions in parallel.

Abstract

Temporal-difference (TD) learning is highly effective at controlling and evaluating an agent's long-term outcomes. Most approaches in this paradigm implement a semi-gradient update to boost the learning speed, which consists of ignoring the gradient of the bootstrapped estimate. While popular, this type of update is prone to divergence, as Baird's counterexample illustrates. Gradient TD methods were introduced to overcome this issue, but have not been widely used, potentially due to issues with learning speed compared to semi-gradient methods. Recently, iterated TD learning was developed to increase the learning speed of TD methods. For that, it learns a sequence of action-value functions in parallel, where each function is optimized to represent the application of the Bellman operator over the previous function in the sequence. While promising, this algorithm can be unstable due to its semi-gradient nature, as each function tracks a moving target. In this work, we modify iterated TD learning by computing the gradients over those moving targets, aiming to build a powerful gradient TD method that competes with semi-gradient methods. Our evaluation reveals that this algorithm, called Gradient Iterated Temporal-Difference learning, has a competitive learning speed against semi-gradient methods across various benchmarks, including Atari games, a result that no prior work on gradient TD methods has demonstrated.