CURL: Contrastive Unsupervised Representations for Reinforcement Learning

TL;DR

CURL introduces a contrastive learning approach to extract high-level features from raw pixel inputs, enabling reinforcement learning agents to achieve superior performance on complex tasks with high sample efficiency.

Contribution

This paper presents CURL, a novel contrastive unsupervised representation learning method that improves pixel-based reinforcement learning by extracting effective features for control tasks.

Findings

Outperforms prior pixel-based methods on DeepMind Control and Atari benchmarks.

Achieves 1.9x and 1.2x performance gains at 100K environment steps.

Nearly matches state-based methods' sample efficiency on DeepMind Control.

Abstract

We present CURL: Contrastive Unsupervised Representations for Reinforcement Learning. CURL extracts high-level features from raw pixels using contrastive learning and performs off-policy control on top of the extracted features. CURL outperforms prior pixel-based methods, both model-based and model-free, on complex tasks in the DeepMind Control Suite and Atari Games showing 1.9x and 1.2x performance gains at the 100K environment and interaction steps benchmarks respectively. On the DeepMind Control Suite, CURL is the first image-based algorithm to nearly match the sample-efficiency of methods that use state-based features. Our code is open-sourced and available at https://github.com/MishaLaskin/curl.