Learning sparse representations in reinforcement learning

TL;DR

This paper explores how sparse internal representations, inspired by neural lateral inhibition, improve reinforcement learning efficiency and success in complex control tasks by reducing interference and supporting generalization.

Contribution

It introduces the hypothesis that sparse representations, akin to cortical lateral inhibition, enhance TD learning, and demonstrates this through computational simulations on classic control problems.

Findings

Sparse representations improve learning speed and success.

Lateral inhibition-inspired sparsity prevents catastrophic interference.

Simulations show better performance on control tasks.

Abstract

Reinforcement learning (RL) algorithms allow artificial agents to improve their selection of actions to increase rewarding experiences in their environments. Temporal Difference (TD) Learning -- a model-free RL method -- is a leading account of the midbrain dopamine system and the basal ganglia in reinforcement learning. These algorithms typically learn a mapping from the agent's current sensed state to a selected action (known as a policy function) via learning a value function (expected future rewards). TD Learning methods have been very successful on a broad range of control tasks, but learning can become intractably slow as the state space of the environment grows. This has motivated methods that learn internal representations of the agent's state, effectively reducing the size of the state space and restructuring state representations in order to support generalization. However, TD Learning coupled with an artificial neural network, as a function approximator, has been shown to fail to learn some fairly simple control tasks, challenging this explanation of reward-based learning. We hypothesize that such failures do not arise in the brain because of the ubiquitous presence of lateral inhibition in the cortex, producing sparse distributed internal representations that support the learning of expected future reward. The sparse conjunctive representations can avoid catastrophic interference while still supporting generalization. We provide support for this conjecture through computational simulations, demonstrating the benefits of learned sparse representations for three problematic classic control tasks: Puddle-world, Mountain-car, and Acrobot.