A Dantzig Selector Approach to Temporal Difference Learning

TL;DR

This paper introduces a novel algorithm combining LSTD with the Dantzig Selector to improve high-dimensional value function approximation in reinforcement learning, addressing limitations of existing regularization methods.

Contribution

It presents a new integration of LSTD with the Dantzig Selector, offering an alternative regularization approach for high-dimensional TD learning.

Findings

Demonstrates improved feature selection in high-dimensional settings

Shows the proposed method addresses drawbacks of L1-regularization approaches

Provides theoretical analysis of the algorithm's performance

Abstract

LSTD is a popular algorithm for value function approximation. Whenever the number of features is larger than the number of samples, it must be paired with some form of regularization. In particular, L1-regularization methods tend to perform feature selection by promoting sparsity, and thus, are well-suited for high-dimensional problems. However, since LSTD is not a simple regression algorithm, but it solves a fixed--point problem, its integration with L1-regularization is not straightforward and might come with some drawbacks (e.g., the P-matrix assumption for LASSO-TD). In this paper, we introduce a novel algorithm obtained by integrating LSTD with the Dantzig Selector. We investigate the performance of the proposed algorithm and its relationship with the existing regularized approaches, and show how it addresses some of their drawbacks.