A Finite-Time Analysis of TD Learning with Linear Function Approximation without Projections or Strong Convexity

TL;DR

This paper provides a new finite-time convergence analysis of projection-free TD learning with linear function approximation, demonstrating it converges without artificial boundedness assumptions even under Markovian noise.

Contribution

It introduces a refined analysis showing projection-free TD learning converges with a specific rate without relying on strong convexity or artificial projections.

Findings

Convergence rate of rac{{ heta^*}||^2_2}{\u007sqrt{T}}

Convergence holds even with Markovian noise

Establishes a self-bounding property of TD updates

Abstract

We investigate the finite-time convergence properties of Temporal Difference (TD) learning with linear function approximation, a cornerstone algorithm in the field of reinforcement learning. We are interested in the so-called ``robust'' setting, where the convergence guarantee does not depend on the minimal curvature of the potential function. While prior work has established convergence guarantees in this setting, these results typically rely on the assumption that each iterate is projected onto a bounded set, a condition that is both artificial and does not match the current practice. In this paper, we challenge the necessity of such an assumption and present a refined analysis of TD learning. For the first time, we show that the simple projection-free variant converges with a rate of $\widetilde{\mathcal{O}}(\frac{||\theta^*||^2_2}{\sqrt{T}})$, even in the presence of Markovian noise. Our analysis reveals a novel self-bounding property of the TD updates and exploits it to guarantee bounded iterates.