Sample-efficient Reinforcement Learning Representation Learning with Curiosity Contrastive Forward Dynamics Model

TL;DR

This paper introduces CCFDM, a sample-efficient reinforcement learning framework that uses contrastive learning and intrinsic rewards from a forward dynamics model to improve exploration, generalization, and performance on pixel-based tasks.

Contribution

The paper proposes CCFDM, a novel framework combining contrastive learning with a forward dynamics model to enhance sample efficiency and exploration in pixel-based RL.

Findings

CCFDM outperforms prior pixel-based RL methods on DeepMind Control Suite.

Intrinsic rewards from FDM prediction error improve exploration.

Contrastive learning enhances generalization in RL agents.

Abstract

Developing an agent in reinforcement learning (RL) that is capable of performing complex control tasks directly from high-dimensional observation such as raw pixels is yet a challenge as efforts are made towards improving sample efficiency and generalization. This paper considers a learning framework for Curiosity Contrastive Forward Dynamics Model (CCFDM) in achieving a more sample-efficient RL based directly on raw pixels. CCFDM incorporates a forward dynamics model (FDM) and performs contrastive learning to train its deep convolutional neural network-based image encoder (IE) to extract conducive spatial and temporal information for achieving a more sample efficiency for RL. In addition, during training, CCFDM provides intrinsic rewards, produced based on FDM prediction error, encourages the curiosity of the RL agent to improve exploration. The diverge and less-repetitive observations provide by both our exploration strategy and data augmentation available in contrastive learning improve not only the sample efficiency but also the generalization. Performance of existing model-free RL methods such as Soft Actor-Critic built on top of CCFDM outperforms prior state-of-the-art pixel-based RL methods on the DeepMind Control Suite benchmark.