On TD(0) with function approximation: Concentration bounds and a centered variant with exponential convergence

TL;DR

This paper derives non-asymptotic convergence bounds for TD(0) with linear function approximation, introduces a centered variant with exponential convergence, and discusses the impact of step-size choices and stationary distribution knowledge.

Contribution

It provides the first non-asymptotic bounds for TD(0), analyzes the effect of step-size and stationary distribution knowledge, and proposes a centered variant with exponential convergence.

Findings

A step-size inversely proportional to iterations cannot guarantee optimal convergence without stationary distribution knowledge.

Averages of TD(0) iterates achieve optimal convergence rates without step-size dependency.

A centered TD(0) variant exhibits exponential convergence in expectation.

Abstract

We provide non-asymptotic bounds for the well-known temporal difference learning algorithm TD(0) with linear function approximators. These include high-probability bounds as well as bounds in expectation. Our analysis suggests that a step-size inversely proportional to the number of iterations cannot guarantee optimal rate of convergence unless we assume (partial) knowledge of the stationary distribution for the Markov chain underlying the policy considered. We also provide bounds for the iterate averaged TD(0) variant, which gets rid of the step-size dependency while exhibiting the optimal rate of convergence. Furthermore, we propose a variant of TD(0) with linear approximators that incorporates a centering sequence, and establish that it exhibits an exponential rate of convergence in expectation. We demonstrate the usefulness of our bounds on two synthetic experimental settings.