Predictor-Corrector(PC) Temporal Difference(TD) Learning (PCTD)

TL;DR

This paper introduces Predictor-Corrector Temporal Difference (PCTD), a novel RL algorithm inspired by numerical ODE methods, which improves approximation accuracy and reduces error in value function estimation compared to traditional TD methods.

Contribution

It proposes a new class of TD learning algorithms based on predictor-corrector methods from numerical analysis, providing both causal and non-causal implementations with improved error bounds.

Findings

PCTD algorithms show reduced Taylor Series error in parameter approximation.

Simulation results demonstrate PCTD's superior performance over TD(0) in infinite horizon tasks.

Both causal and non-causal PCTD implementations are viable and effective.

Abstract

Using insight from numerical approximation of ODEs and the problem formulation and solution methodology of TD learning through a Galerkin relaxation, I propose a new class of TD learning algorithms. After applying the improved numerical methods, the parameter being approximated has a guaranteed order of magnitude reduction in the Taylor Series error of the solution to the ODE for the parameter $\theta(t)$ that is used in constructing the linearly parameterized value function. Predictor-Corrector Temporal Difference (PCTD) is what I call the translated discrete time Reinforcement Learning(RL) algorithm from the continuous time ODE using the theory of Stochastic Approximation(SA). Both causal and non-causal implementations of the algorithm are provided, and simulation results are listed for an infinite horizon task to compare the original TD(0) algorithm against both versions of PCTD(0).