Accelerating Representation Learning with View-Consistent Dynamics in Data-Efficient Reinforcement Learning

TL;DR

This paper introduces View-Consistent Dynamics (VCD), a novel method that accelerates representation learning in deep RL by enforcing view-consistency on dynamics, leading to state-of-the-art data efficiency in visual control tasks.

Contribution

The paper proposes a formal Multi-view Markov Decision Process and a view-consistent dynamics model to improve data efficiency in visual RL tasks.

Findings

VCD achieves state-of-the-art data efficiency on DeepMind Control Suite.

VCD outperforms existing methods on Atari-100k.

Enforcing view-consistency improves representation learning in RL.

Abstract

Learning informative representations from image-based observations is of fundamental concern in deep Reinforcement Learning (RL). However, data-inefficiency remains a significant barrier to this objective. To overcome this obstacle, we propose to accelerate state representation learning by enforcing view-consistency on the dynamics. Firstly, we introduce a formalism of Multi-view Markov Decision Process (MMDP) that incorporates multiple views of the state. Following the structure of MMDP, our method, View-Consistent Dynamics (VCD), learns state representations by training a view-consistent dynamics model in the latent space, where views are generated by applying data augmentation to states. Empirical evaluation on DeepMind Control Suite and Atari-100k demonstrates VCD to be the SoTA data-efficient algorithm on visual control tasks.