Reconstruction or Semantics? What Makes a Latent Space Useful for Robotic World Models

TL;DR

This paper evaluates different latent spaces for robotic world models using diffusion models, finding semantic encoders outperform reconstruction-based ones in policy-relevant tasks.

Contribution

It systematically compares six encoders for latent diffusion models and introduces axes for assessing world model performance in robotics.

Findings

Semantic encoders outperform reconstruction encoders in policy tasks.

Visual fidelity alone is not sufficient for world model effectiveness.

Semantic latent spaces provide a stronger foundation for robotics diffusion models.

Abstract

World model-based policy evaluation is a practical proxy for testing real-world robot control by rolling out candidate actions in action-conditioned video diffusion models. As these models increasingly adopt latent diffusion modeling (LDM), choosing the right latent space becomes critical. While the status quo uses autoencoding latent spaces like VAEs that are primarily trained for pixel reconstruction, recent work suggests benefits from pretrained encoders with representation-aligned semantic latent spaces. We systematically evaluate these latent spaces for action-conditioned LDM by comparing six reconstruction and semantic encoders to train world model variants under a fixed protocol on BridgeV2 dataset, and show effective world model training in high-dimensional representation spaces with and without dimension compression. We then propose three axes to assess robotic world model performance: visual fidelity, planning and downstream policy performance, and latent representation quality. Our results show visual fidelity alone is insufficient for world model selection. While reconstruction encoders like VAE and Cosmos achieve strong pixel-level scores, semantic encoders such as V-JEPA 2.1 (strongest overall on policy), Web-DINO, and SigLIP 2 generally excel across the other two axes at all model scales. Our study advocates semantic latent space as stronger foundation for policy-relevant robotics diffusion world models.