Q($\lambda$) with Off-Policy Corrections

TL;DR

This paper introduces an alternative off-policy TD($5$) method that uses current Q-function-based reward corrections, providing convergence guarantees and empirical validation in continuous control tasks.

Contribution

It proposes a novel off-policy correction approach using current Q-values for rewards, with theoretical convergence analysis and empirical demonstration.

Findings

Convergence achieved under specific policy and parameter conditions

Theoretical relationship between policy divergence, trace parameter, and discount factor

Empirical validation on a continuous-state control task

Abstract

We propose and analyze an alternate approach to off-policy multi-step temporal difference learning, in which off-policy returns are corrected with the current Q-function in terms of rewards, rather than with the target policy in terms of transition probabilities. We prove that such approximate corrections are sufficient for off-policy convergence both in policy evaluation and control, provided certain conditions. These conditions relate the distance between the target and behavior policies, the eligibility trace parameter and the discount factor, and formalize an underlying tradeoff in off-policy TD($\lambda$). We illustrate this theoretical relationship empirically on a continuous-state control task.