Decoupling Representation Learning from Reinforcement Learning

TL;DR

This paper introduces a novel unsupervised learning task called Augmented Temporal Contrast (ATC) to improve representation learning in deep reinforcement learning from images, decoupling it from policy learning and achieving superior performance.

Contribution

It proposes ATC, a new contrastive learning method for image representations in RL, and demonstrates its effectiveness across various benchmarks and tasks.

Findings

ATC-trained encoders outperform end-to-end RL encoders.

Pre-trained encoders with ATC improve RL agent performance.

Multi-task encoders generalize across different environments.

Abstract

In an effort to overcome limitations of reward-driven feature learning in deep reinforcement learning (RL) from images, we propose decoupling representation learning from policy learning. To this end, we introduce a new unsupervised learning (UL) task, called Augmented Temporal Contrast (ATC), which trains a convolutional encoder to associate pairs of observations separated by a short time difference, under image augmentations and using a contrastive loss. In online RL experiments, we show that training the encoder exclusively using ATC matches or outperforms end-to-end RL in most environments. Additionally, we benchmark several leading UL algorithms by pre-training encoders on expert demonstrations and using them, with weights frozen, in RL agents; we find that agents using ATC-trained encoders outperform all others. We also train multi-task encoders on data from multiple environments and show generalization to different downstream RL tasks. Finally, we ablate components of ATC, and introduce a new data augmentation to enable replay of (compressed) latent images from pre-trained encoders when RL requires augmentation. Our experiments span visually diverse RL benchmarks in DeepMind Control, DeepMind Lab, and Atari, and our complete code is available at https://github.com/astooke/rlpyt/tree/master/rlpyt/ul.