Contrastive Unsupervised Learning of World Model with Invariant Causal Features

TL;DR

This paper introduces a contrastive unsupervised learning approach to develop a world model that captures invariant causal features, improving out-of-distribution generalization and sim-to-real transfer in reinforcement learning tasks.

Contribution

It proposes a novel intervention invariant auxiliary task using depth prediction and data augmentation to learn invariant causal features in an unsupervised manner.

Findings

Outperforms state-of-the-art in out-of-distribution point navigation

Enables effective sim-to-real transfer of perception modules

Performs comparably on DeepMind control suite without explicit depth

Abstract

In this paper we present a world model, which learns causal features using the invariance principle. In particular, we use contrastive unsupervised learning to learn the invariant causal features, which enforces invariance across augmentations of irrelevant parts or styles of the observation. The world-model-based reinforcement learning methods independently optimize representation learning and the policy. Thus naive contrastive loss implementation collapses due to a lack of supervisory signals to the representation learning module. We propose an intervention invariant auxiliary task to mitigate this issue. Specifically, we utilize depth prediction to explicitly enforce the invariance and use data augmentation as style intervention on the RGB observation space. Our design leverages unsupervised representation learning to learn the world model with invariant causal features. Our proposed method significantly outperforms current state-of-the-art model-based and model-free reinforcement learning methods on out-of-distribution point navigation tasks on the iGibson dataset. Moreover, our proposed model excels at the sim-to-real transfer of our perception learning module. Finally, we evaluate our approach on the DeepMind control suite and enforce invariance only implicitly since depth is not available. Nevertheless, our proposed model performs on par with the state-of-the-art counterpart.