$\ell_1$ Regularized Gradient Temporal-Difference Learning

TL;DR

This paper introduces $ ext{L}_1$ regularized Gradient TD algorithms for stable, efficient off-policy learning with large feature sets, combining regularization with convergence guarantees and empirical validation.

Contribution

It develops a new family of $ ext{L}_1$ regularized GTD algorithms using soft thresholding, addressing overfitting and computational issues in high-dimensional settings.

Findings

Algorithms converge under certain conditions.

Regularization improves feature selection and stability.

Numerical experiments demonstrate effectiveness.

Abstract

In this paper, we study the Temporal Difference (TD) learning with linear value function approximation. It is well known that most TD learning algorithms are unstable with linear function approximation and off-policy learning. Recent development of Gradient TD (GTD) algorithms has addressed this problem successfully. However, the success of GTD algorithms requires a set of well chosen features, which are not always available. When the number of features is huge, the GTD algorithms might face the problem of overfitting and being computationally expensive. To cope with this difficulty, regularization techniques, in particular $\ell_1$ regularization, have attracted significant attentions in developing TD learning algorithms. The present work combines the GTD algorithms with $\ell_1$ regularization. We propose a family of $\ell_1$ regularized GTD algorithms, which employ the well known soft thresholding operator. We investigate convergence properties of the proposed algorithms, and depict their performance with several numerical experiments.