Regularized Centered Emphatic Temporal Difference Learning

TL;DR

This paper introduces Regularized Emphatic TD (RETD), a novel off-policy TD learning method that stabilizes emphatic learning through targeted regularization, improving convergence and robustness.

Contribution

The paper proposes RETD, which preserves emphatic projection geometry while ensuring convergence by regularizing the centering recursion, addressing instability issues.

Findings

RETD avoids instability of naive centered emphatic learning.

RETD maintains favorable emphatic geometry.

RETD exhibits robustness across a range of regularization parameters.

Abstract

Off-policy temporal-difference (TD) learning with function approximation faces a structural tradeoff among stability, projection geometry, and variance control. Emphatic TD (ETD) improves the off-policy projection geometry through follow-on emphasis, but the follow-on trace can have high variance. We revisit this tradeoff through Bellman-error centering. Although centering naturally removes a common drift term from TD errors, we show that a naive centered emphatic extension introduces an auxiliary coupling that can destroy the positive-definiteness of the ETD key matrix. We propose \emph{Regularized Emphatic Temporal-Difference Learning} (RETD), which preserves the follow-on trace and regularizes only the auxiliary centering recursion, corresponding to lifting the lower-right block of the coupled key matrix from \(1\) to \(1+c\). We derive the RETD core matrix, prove convergence under a conservative sufficient regularization condition, and evaluate the method on diagnostic linear off-policy prediction tasks. The experiments show that RETD avoids the instability of naive centered emphatic learning, preserves favorable emphatic geometry, and exhibits a robust intermediate regime for the regularization parameter \(c\) across the diagnostics.