ReCoRe: Regularized Contrastive Representation Learning of World Model

TL;DR

ReCoRe introduces a regularized contrastive learning approach for world models in reinforcement learning, enhancing sample efficiency and generalization in visual navigation tasks, especially under appearance variations.

Contribution

It proposes an intervention-invariant regularizer combined with contrastive learning to improve invariant feature learning in world models for RL.

Findings

Outperforms state-of-the-art RL methods on iGibson benchmark

Achieves superior out-of-distribution point navigation performance

Demonstrates effective sim-to-real transfer of perception module

Abstract

While recent model-free Reinforcement Learning (RL) methods have demonstrated human-level effectiveness in gaming environments, their success in everyday tasks like visual navigation has been limited, particularly under significant appearance variations. This limitation arises from (i) poor sample efficiency and (ii) over-fitting to training scenarios. To address these challenges, we present a world model that learns invariant features using (i) contrastive unsupervised learning and (ii) an intervention-invariant regularizer. Learning an explicit representation of the world dynamics i.e. a world model, improves sample efficiency while contrastive learning implicitly enforces learning of invariant features, which improves generalization. However, the na\"ive integration of contrastive loss to world models is not good enough, as world-model-based RL methods independently optimize representation learning and agent policy. To overcome this issue, we propose an intervention-invariant regularizer in the form of an auxiliary task such as depth prediction, image denoising, image segmentation, etc., that explicitly enforces invariance to style interventions. Our method outperforms current state-of-the-art model-based and model-free RL methods and significantly improves on out-of-distribution point navigation tasks evaluated on the iGibson benchmark. With only visual observations, we further demonstrate that our approach outperforms recent language-guided foundation models for point navigation, which is essential for deployment on robots with limited computation capabilities. Finally, we demonstrate that our proposed model excels at the sim-to-real transfer of its perception module on the Gibson benchmark.