A Finite Time Analysis of Temporal Difference Learning With Linear Function Approximation

TL;DR

This paper provides a clear finite-time analysis of TD learning with linear function approximation, enhancing understanding of its statistical efficiency and extending results to TD($$) and high-dimensional Q-learning.

Contribution

It offers the first simple, explicit finite-time bounds for TD with linear approximation, using standard stochastic gradient techniques, and extends these results to TD($$) and Q-learning in complex settings.

Findings

Finite-time bounds for TD with linear function approximation.

Extension of analysis to TD($) and high-dimensional Q-learning.

Demonstrates the efficiency and applicability of the proposed analysis.

Abstract

Temporal difference learning (TD) is a simple iterative algorithm used to estimate the value function corresponding to a given policy in a Markov decision process. Although TD is one of the most widely used algorithms in reinforcement learning, its theoretical analysis has proved challenging and few guarantees on its statistical efficiency are available. In this work, we provide a simple and explicit finite time analysis of temporal difference learning with linear function approximation. Except for a few key insights, our analysis mirrors standard techniques for analyzing stochastic gradient descent algorithms, and therefore inherits the simplicity and elegance of that literature. Final sections of the paper show how all of our main results extend to the study of TD learning with eligibility traces, known as TD($\lambda$), and to Q-learning applied in high-dimensional optimal stopping problems.