Integrating Contrastive Learning with Dynamic Models for Reinforcement Learning from Images

TL;DR

This paper introduces a novel self-supervised learning approach that combines contrastive learning with dynamic models to improve sample efficiency and generalization in reinforcement learning from images.

Contribution

It proposes a method integrating contrastive learning with nonlinear transition models to enhance Markovianity and invariance in learned embeddings.

Findings

Achieves higher sample efficiency on Deepmind control suite

Demonstrates improved generalization over state-of-the-art contrastive methods

Effectively combines contrastive learning with dynamic modeling

Abstract

Recent methods for reinforcement learning from images use auxiliary tasks to learn image features that are used by the agent's policy or Q-function. In particular, methods based on contrastive learning that induce linearity of the latent dynamics or invariance to data augmentation have been shown to greatly improve the sample efficiency of the reinforcement learning algorithm and the generalizability of the learned embedding. We further argue, that explicitly improving Markovianity of the learned embedding is desirable and propose a self-supervised representation learning method which integrates contrastive learning with dynamic models to synergistically combine these three objectives: (1) We maximize the InfoNCE bound on the mutual information between the state- and action-embedding and the embedding of the next state to induce a linearly predictive embedding without explicitly learning a linear transition model, (2) we further improve Markovianity of the learned embedding by explicitly learning a non-linear transition model using regression, and (3) we maximize the mutual information between the two nonlinear predictions of the next embeddings based on the current action and two independent augmentations of the current state, which naturally induces transformation invariance not only for the state embedding, but also for the nonlinear transition model. Experimental evaluation on the Deepmind control suite shows that our proposed method achieves higher sample efficiency and better generalization than state-of-art methods based on contrastive learning or reconstruction.