Schedule Based Temporal Difference Algorithms

TL;DR

This paper introduces a flexible schedule-based extension to TD($ lambda$) algorithms, allowing the user to specify how weights are assigned to different n-step returns over time, with proven convergence.

Contribution

It proposes a novel lambda-schedule procedure for TD algorithms, enabling dynamic weight assignment and providing convergence guarantees for on-policy and off-policy variants.

Findings

The lambda-schedule generalizes TD($ lambda$) with time-varying parameters.

Proposed algorithms converge almost surely under general conditions.

Flexible weight assignment improves adaptability of TD methods.

Abstract

Learning the value function of a given policy from data samples is an important problem in Reinforcement Learning. TD($\lambda$) is a popular class of algorithms to solve this problem. However, the weights assigned to different $n$-step returns in TD($\lambda$), controlled by the parameter $\lambda$, decrease exponentially with increasing $n$. In this paper, we present a $\lambda$-schedule procedure that generalizes the TD($\lambda$) algorithm to the case when the parameter $\lambda$ could vary with time-step. This allows flexibility in weight assignment, i.e., the user can specify the weights assigned to different $n$-step returns by choosing a sequence $\{\lambda_t\}_{t \geq 1}$. Based on this procedure, we propose an on-policy algorithm - TD($\lambda$)-schedule, and two off-policy algorithms - GTD($\lambda$)-schedule and TDC($\lambda$)-schedule, respectively. We provide proofs of almost sure convergence for all three algorithms under a general Markov noise framework.