Optimal Control with Natural Images: Efficient Reinforcement Learning using Overcomplete Sparse Codes

TL;DR

This paper demonstrates that using overcomplete sparse codes for natural images enables efficient reinforcement learning for optimal control tasks, scaling to larger problems without deep learning.

Contribution

It introduces a new benchmark and theoretical insights showing overcomplete sparse coding facilitates scalable optimal control with natural images.

Findings

Overcomplete sparse codes enable solving larger control tasks.

Reinforcement learning with efficient representations scales well to long horizons.

Deep learning is not required for effective control with natural images.

Abstract

Optimal control and sequential decision making are widely used in many complex tasks. Optimal control over a sequence of natural images is a first step towards understanding the role of vision in control. Here, we formalize this problem as a reinforcement learning task, and derive general conditions under which an image includes enough information to implement an optimal policy. Reinforcement learning is shown to provide a computationally efficient method for finding optimal policies when natural images are encoded into "efficient" image representations. This is demonstrated by introducing a new reinforcement learning benchmark that easily scales to large numbers of states and long horizons. In particular, by representing each image as an overcomplete sparse code, we are able to efficiently solve an optimal control task that is orders of magnitude larger than those tasks solvable using complete codes. Theoretical justification for this behaviour is provided. This work also demonstrates that deep learning is not necessary for efficient optimal control with natural images.