Reward-Predictive Clustering

TL;DR

This paper introduces a clustering algorithm that enables reward-predictive state abstractions to be used in deep learning, significantly improving learning speed and transferability in high-dimensional reinforcement learning tasks.

Contribution

It extends reward-predictive state abstractions from tabular to deep learning settings with a new clustering algorithm and provides theoretical and empirical validation.

Findings

Deep reward-predictive networks compress inputs effectively.

Significant acceleration of learning in visual control tasks.

Pre-trained representations can be reused for transfer learning.

Abstract

Recent advances in reinforcement-learning research have demonstrated impressive results in building algorithms that can out-perform humans in complex tasks. Nevertheless, creating reinforcement-learning systems that can build abstractions of their experience to accelerate learning in new contexts still remains an active area of research. Previous work showed that reward-predictive state abstractions fulfill this goal, but have only be applied to tabular settings. Here, we provide a clustering algorithm that enables the application of such state abstractions to deep learning settings, providing compressed representations of an agent's inputs that preserve the ability to predict sequences of reward. A convergence theorem and simulations show that the resulting reward-predictive deep network maximally compresses the agent's inputs, significantly speeding up learning in high dimensional visual control tasks. Furthermore, we present different generalization experiments and analyze under which conditions a pre-trained reward-predictive representation network can be re-used without re-training to accelerate learning -- a form of systematic out-of-distribution transfer.