DiLA: Disentangled Latent Action World Models

TL;DR

DiLA introduces a disentangled latent action world model that balances action abstraction with high-quality video generation, advancing self-supervised world modeling.

Contribution

The paper presents DiLA, a novel framework that co-evolves disentanglement and latent action learning to improve video generation and action interpretability.

Findings

DiLA achieves superior video generation quality.

It enables effective action transfer and visual planning.

The model offers enhanced manifold interpretability.

Abstract

Latent Action Models (LAMs) enable the learning of world models from unlabeled video by inferring abstract actions between consecutive frames. However, LAMs face a fundamental trade-off between action abstraction and generation fidelity. Existing methods typically circumvent this issue by using two-stage training with pre-trained world models or by limiting predictions to optical flow. In this paper, we introduce DiLA, a novel Disentangled Latent Action world model that aims to resolve this trade-off via content-structure disentanglement. Our key insight is that disentanglement and latent action learning are co-evolving: the predictive bottleneck inherent in latent action learning serves as a driving force for disentanglement, compelling the model to distill spatial layouts into the structure pathway while offloading visual details to a separate content pathway for generation. This synergy yields a continuous, semantically structured latent action space without compromising generative quality. DiLA achieves superior results in video generation quality, action transfer, visual planning, and manifold interpretability. These findings establish DiLA as a unified framework that simultaneously achieves high-level action abstraction and high-fidelity generation, advancing the frontier of self-supervised world model learning.