MAD: Motion Appearance Decoupling for efficient Driving World Models

TL;DR

This paper introduces MAD, a two-stage framework that decouples motion learning from appearance synthesis to efficiently adapt generalist video diffusion models into controllable driving world models with minimal supervision.

Contribution

It presents a novel decoupling approach that separates motion prediction from appearance rendering, enabling efficient adaptation of video models for autonomous driving tasks.

Findings

Achieves comparable performance to state-of-the-art models with less than 6% of the compute.

Outperforms open-source competitors on the LTX benchmark.

Supports diverse controls including text, ego, and object manipulation.

Abstract

Recent video diffusion models generate photorealistic, temporally coherent videos, yet they fall short as reliable world models for autonomous driving, where structured motion and physically consistent interactions are essential. Adapting these generalist video models to driving domains has shown promise but typically requires massive domain-specific data and costly fine-tuning. We propose an efficient adaptation framework that converts generalist video diffusion models into controllable driving world models with minimal supervision. The key idea is to decouple motion learning from appearance synthesis. First, the model is adapted to predict structured motion in a simplified form: videos of skeletonized agents and scene elements, focusing learning on physical and social plausibility. Then, the same backbone is reused to synthesize realistic RGB videos conditioned on these motion sequences, effectively "dressing" the motion with texture and lighting. This two-stage process mirrors a reasoning-rendering paradigm: first infer dynamics, then render appearance. Our experiments show this decoupled approach is exceptionally efficient: adapting SVD, we match prior SOTA models with less than 6% of their compute. Scaling to LTX, our MAD-LTX model outperforms all open-source competitors, and supports a comprehensive suite of text, ego, and object controls. Project page: https://vita-epfl.github.io/MAD-World-Model/