R-GTD: A Geometric Analysis of Gradient Temporal-Difference Learning in Singular Regimes

TL;DR

This paper introduces R-GTD, a regularized gradient TD learning algorithm that guarantees convergence even when the feature interaction matrix is singular, addressing stability issues in off-policy evaluation.

Contribution

It proposes a reformulated optimization objective leading to R-GTD, which ensures convergence under singular FIM conditions, supported by geometric analysis and empirical validation.

Findings

R-GTD converges to a unique solution with singular FIM.

Theoretical error bounds are established for R-GTD.

Empirical results validate the effectiveness of R-GTD.

Abstract

Gradient temporal-difference (GTD) learning algorithms are widely used for off-policy policy evaluation with function approximation. However, existing convergence analyses rely on the restrictive assumption that the so-called feature interaction matrix (FIM) is nonsingular. In practice, the FIM can become singular and leads to instability or degraded performance. While some prior works have applied regularization to relax the nonsingularity assumption, their theoretical guarantees inevitably rely on other restrictive conditions. In this paper, we propose a regularized optimization objective by reformulating the mean-square projected Bellman error minimization. This formulation naturally yields a regularized GTD algorithms, referred to as R-GTD, which guarantees convergence to a unique solution even when the FIM is singular. We conduct a geometric analysis to establish theoretical convergence guarantees and explicit error bounds for the proposed method, and validate its effectiveness through empirical experiments.