Incrementally Learning Functions of the Return

TL;DR

This paper introduces a method for incrementally learning functions of the return in reinforcement learning by estimating its moments with a modified TD algorithm, enabling approximation of complex functions of the return.

Contribution

It proposes a novel approach to estimate functions of the return through moments, extending traditional TD methods for broader function approximation in RL.

Findings

Moments of the return can be learned online using a modified TD algorithm.

Functions of the return can be approximated via Taylor expansion using these moments.

The method broadens the scope of value function estimation in reinforcement learning.

Abstract

Temporal difference methods enable efficient estimation of value functions in reinforcement learning in an incremental fashion, and are of broader interest because they correspond learning as observed in biological systems. Standard value functions correspond to the expected value of a sum of discounted returns. While this formulation is often sufficient for many purposes, it would often be useful to be able to represent functions of the return as well. Unfortunately, most such functions cannot be estimated directly using TD methods. We propose a means of estimating functions of the return using its moments, which can be learned online using a modified TD algorithm. The moments of the return are then used as part of a Taylor expansion to approximate analytic functions of the return.