VDAWorld: World Modelling via VLM-Directed Abstraction and Simulation

TL;DR

VDAWorld introduces a novel world modeling approach that uses a vision-language model to create and simulate abstract scene representations, overcoming limitations of traditional generative video models.

Contribution

The paper presents VDAWorld, a framework where a VLM orchestrates scene abstraction and simulation, enabling structured, queryable, and physically consistent world models.

Findings

Enables high-quality dynamic scene simulation

Constructs grounded scene representations from image caption pairs

Adapts physics simulation to scene content

Abstract

Generative video models, a leading approach to world modeling, face fundamental limitations. They often violate physical and logical rules, lack interactivity, and operate as opaque black boxes ill-suited for building structured, queryable worlds. To overcome these challenges, we propose a new paradigm focused on distilling an image caption pair into a tractable, abstract representation optimized for simulation. We introduce VDAWorld, a framework where a Vision-Language Model (VLM) acts as an intelligent agent to orchestrate this process. The VLM autonomously constructs a grounded (2D or 3D) scene representation by selecting from a suite of vision tools, and accordingly chooses a compatible physics simulator (e.g., rigid body, fluid) to act upon it. VDAWorld can then infer latent dynamics from the static scene to predict plausible future states. Our experiments show that this combination of intelligent abstraction and adaptive simulation results in a versatile world model capable of producing high quality simulations across a wide range of dynamic scenarios.