On a convergent off -policy temporal difference learning algorithm in on-line learning environment

TL;DR

This paper rigorously analyzes the convergence of an off-policy temporal difference learning algorithm with linear function approximation in online environments, supported by empirical results on standard counterexamples.

Contribution

It provides a formal convergence proof for the TDC algorithm with importance weighting in online settings, which was previously lacking.

Findings

The TDC algorithm converges under certain conditions.

Empirical results validate theoretical convergence on standard counterexamples.

The analysis demonstrates linear per-step computational complexity.

Abstract

In this paper we provide a rigorous convergence analysis of a "off"-policy temporal difference learning algorithm with linear function approximation and per time-step linear computational complexity in "online" learning environment. The algorithm considered here is TDC with importance weighting introduced by Maei et al. We support our theoretical results by providing suitable empirical results for standard off-policy counterexamples.