Properties of the Least Squares Temporal Difference learning algorithm

TL;DR

This paper provides multiple theoretical perspectives on the LSTD algorithm, compares it with Bellman Residual Minimization, and discusses regularization and episodic adaptations, deepening understanding of its properties.

Contribution

It introduces four different conceptual frameworks for analyzing LSTD, compares its optimization with BRM, and explores regularization and episodic modifications.

Findings

Multiple insights into LSTD through operator, statistical, dynamical, and geometric views.

Comparison showing differences between LSTD and Bellman Residual Minimization.

Discussion of regularization techniques and episodic process adaptations.

Abstract

This paper presents four different ways of looking at the well-known Least Squares Temporal Differences (LSTD) algorithm for computing the value function of a Markov Reward Process, each of them leading to different insights: the operator-theory approach via the Galerkin method, the statistical approach via instrumental variables, the linear dynamical system view as well as the limit of the TD iteration. We also give a geometric view of the algorithm as an oblique projection. Furthermore, there is an extensive comparison of the optimization problem solved by LSTD as compared to Bellman Residual Minimization (BRM). We then review several schemes for the regularization of the LSTD solution. We then proceed to treat the modification of LSTD for the case of episodic Markov Reward Processes.