Greedy Algorithms for Sparse Reinforcement Learning

TL;DR

This paper explores greedy algorithms, specifically variants of orthogonal matching pursuit, for feature selection in sparse reinforcement learning, demonstrating promising theoretical guarantees and empirical performance improvements over existing methods.

Contribution

It introduces and analyzes new greedy algorithms for sparse RL, providing theoretical insights and empirical evidence of their effectiveness compared to $L_1$ regularization approaches.

Findings

OMP-BRM offers theoretical guarantees under certain conditions.

OMP-TD outperforms prior methods in accuracy and efficiency.

Natural sparse recovery scenarios may fail, but variants like OMP-BRM and OMP-TD show promise.

Abstract

Feature selection and regularization are becoming increasingly prominent tools in the efforts of the reinforcement learning (RL) community to expand the reach and applicability of RL. One approach to the problem of feature selection is to impose a sparsity-inducing form of regularization on the learning method. Recent work on $L_1$ regularization has adapted techniques from the supervised learning literature for use with RL. Another approach that has received renewed attention in the supervised learning community is that of using a simple algorithm that greedily adds new features. Such algorithms have many of the good properties of the $L_1$ regularization methods, while also being extremely efficient and, in some cases, allowing theoretical guarantees on recovery of the true form of a sparse target function from sampled data. This paper considers variants of orthogonal matching pursuit (OMP) applied to reinforcement learning. The resulting algorithms are analyzed and compared experimentally with existing $L_1$ regularized approaches. We demonstrate that perhaps the most natural scenario in which one might hope to achieve sparse recovery fails; however, one variant, OMP-BRM, provides promising theoretical guarantees under certain assumptions on the feature dictionary. Another variant, OMP-TD, empirically outperforms prior methods both in approximation accuracy and efficiency on several benchmark problems.