Learning Invariant Visual Representations for Planning with Joint-Embedding Predictive World Models

TL;DR

This paper introduces a method to improve the robustness of visual world models in planning tasks by using a bisimulation encoder, which reduces sensitivity to irrelevant visual variations and enables smaller, more efficient latent representations.

Contribution

The paper proposes augmenting latent predictive models with a bisimulation encoder to enhance robustness against slow features and irrelevant visual changes, while reducing latent space size.

Findings

Model improves robustness to background changes and distractors.

Achieves up to 10x smaller latent space than previous models.

Maintains robustness across different pretrained visual encoders.

Abstract

World models learned from high-dimensional visual observations allow agents to make decisions and plan directly in latent space, avoiding pixel-level reconstruction. However, recent latent predictive architectures (JEPAs), including the DINO world model (DINO-WM), display a degradation in test time robustness due to their sensitivity to "slow features". These include visual variations such as background changes and distractors that are irrelevant to the task being solved. We address this limitation by augmenting the predictive objective with a bisimulation encoder that enforces control-relevant state equivalence, mapping states with similar transition dynamics to nearby latent states while limiting contributions from slow features. We evaluate our model on a simple navigation task under different test-time background changes and visual distractors. Across all benchmarks, our model consistently improves robustness to slow features while operating in a reduced latent space, up to 10x smaller than that of DINO-WM. Moreover, our model is agnostic to the choice of pretrained visual encoder and maintains robustness when paired with DINOv2, SimDINOv2, and iBOT features.