A Non-asymptotic Analysis of Non-parametric Temporal-Difference Learning

TL;DR

This paper provides a non-asymptotic convergence analysis of regularized non-parametric TD(0) in RKHS, demonstrating convergence rates and applicability to continuous-state Markov processes.

Contribution

It offers the first explicit non-asymptotic convergence rates for regularized non-parametric TD(0) in RKHS, including in Markovian settings.

Findings

Convergence of averaged iterates to the optimal value function.

Explicit convergence rates depending on the regularity of the value function.

Numerical illustration on a continuous-state Markov reward process.

Abstract

Temporal-difference learning is a popular algorithm for policy evaluation. In this paper, we study the convergence of the regularized non-parametric TD(0) algorithm, in both the independent and Markovian observation settings. In particular, when TD is performed in a universal reproducing kernel Hilbert space (RKHS), we prove convergence of the averaged iterates to the optimal value function, even when it does not belong to the RKHS. We provide explicit convergence rates that depend on a source condition relating the regularity of the optimal value function to the RKHS. We illustrate this convergence numerically on a simple continuous-state Markov reward process.