Investigating the Properties of Neural Network Representations in Reinforcement Learning

TL;DR

This paper empirically investigates the properties of neural network representations in deep reinforcement learning, focusing on how these properties support transfer learning across tasks and environments.

Contribution

It introduces a systematic methodology to analyze and correlate representational properties with transfer performance in deep reinforcement learning agents.

Findings

Certain representational properties correlate with better transfer performance.

Representation quality varies with task similarity and training schemes.

Methodology generalizes across different RL environments and agents.

Abstract

In this paper we investigate the properties of representations learned by deep reinforcement learning systems. Much of the early work on representations for reinforcement learning focused on designing fixed-basis architectures to achieve properties thought to be desirable, such as orthogonality and sparsity. In contrast, the idea behind deep reinforcement learning methods is that the agent designer should not encode representational properties, but rather that the data stream should determine the properties of the representation -- good representations emerge under appropriate training schemes. In this paper we bring these two perspectives together, empirically investigating the properties of representations that support transfer in reinforcement learning. We introduce and measure six representational properties over more than 25 thousand agent-task settings. We consider Deep Q-learning agents with different auxiliary losses in a pixel-based navigation environment, with source and transfer tasks corresponding to different goal locations. We develop a method to better understand why some representations work better for transfer, through a systematic approach varying task similarity and measuring and correlating representation properties with transfer performance. We demonstrate the generality of the methodology by investigating representations learned by a Rainbow agent that successfully transfer across games modes in Atari 2600.