Differential TD Learning for Value Function Approximation

TL;DR

This paper introduces a new differential TD learning algorithm that addresses variance divergence in discounted-cost settings and bias issues in average cost scenarios, showing significant performance improvements.

Contribution

It proposes a novel differential TD method based on gradient representations, improving value function approximation in Markovian models with smooth dynamics.

Findings

Variance reduced by two orders of magnitude in speed scaling applications

Numerical examples demonstrate remarkable performance improvements

Addresses divergence and bias issues in traditional TD learning

Abstract

Value functions arise as a component of algorithms as well as performance metrics in statistics and engineering applications. Computation of the associated Bellman equations is numerically challenging in all but a few special cases. A popular approximation technique is known as Temporal Difference (TD) learning. The algorithm introduced in this paper is intended to resolve two well-known problems with this approach: In the discounted-cost setting, the variance of the algorithm diverges as the discount factor approaches unity. Second, for the average cost setting, unbiased algorithms exist only in special cases. It is shown that the gradient of any of these value functions admits a representation that lends itself to algorithm design. Based on this result, the new differential TD method is obtained for Markovian models on Euclidean space with smooth dynamics. Numerical examples show remarkable improvements in performance. In application to speed scaling, variance is reduced by two orders of magnitude.