Learning Sparse Representations in Reinforcement Learning with Sparse Coding

TL;DR

This paper introduces a supervised sparse coding method for reinforcement learning that guarantees global optimality of local minima and outperforms traditional tile-coding representations in policy evaluation.

Contribution

It develops a novel supervised sparse coding objective with provable properties and demonstrates its effectiveness over existing fixed sparse representations in reinforcement learning.

Findings

Supervised sparse coding outperforms tile-coding in policy evaluation.

All local minima in the proposed objective are global minima.

Supervised approach is more effective than unsupervised sparse coding.

Abstract

A variety of representation learning approaches have been investigated for reinforcement learning; much less attention, however, has been given to investigating the utility of sparse coding. Outside of reinforcement learning, sparse coding representations have been widely used, with non-convex objectives that result in discriminative representations. In this work, we develop a supervised sparse coding objective for policy evaluation. Despite the non-convexity of this objective, we prove that all local minima are global minima, making the approach amenable to simple optimization strategies. We empirically show that it is key to use a supervised objective, rather than the more straightforward unsupervised sparse coding approach. We compare the learned representations to a canonical fixed sparse representation, called tile-coding, demonstrating that the sparse coding representation outperforms a wide variety of tilecoding representations.