Co-Evolving Latent Action World Models

TL;DR

CoLA-World introduces a joint training paradigm for latent action world models, enabling co-evolution of models and improved video simulation and planning performance.

Contribution

It proposes a novel method for jointly training latent action models with pretrained world models, overcoming representational collapse and enhancing control and simulation quality.

Findings

Matches or outperforms prior methods in video simulation quality.

Improves downstream visual planning performance.

Successfully implements co-evolution of models through a warm-up phase.

Abstract

Adapting pretrained video generation models into controllable world models via latent actions is a promising step towards creating generalist world models. The dominant paradigm adopts a two-stage approach that trains latent action model (LAM) and the world model separately, resulting in redundant training and limiting their potential for co-adaptation. A conceptually simple and appealing idea is to directly replace the forward dynamic model in LAM with a powerful world model and training them jointly, but it is non-trivial and prone to representational collapse. In this work, we propose CoLA-World, which for the first time successfully realizes this synergistic paradigm, resolving the core challenge in joint learning through a critical warm-up phase that effectively aligns the representations of the from-scratch LAM with the pretrained world model. This unlocks a co-evolution cycle: the world model acts as a knowledgeable tutor, providing gradients to shape a high-quality LAM, while the LAM offers a more precise and adaptable control interface to the world model. Empirically, CoLA-World matches or outperforms prior two-stage methods in both video simulation quality and downstream visual planning, establishing a robust and efficient new paradigm for the field.