Interactive World Simulator for Robot Policy Training and Evaluation

TL;DR

This paper introduces an Interactive World Simulator that creates fast, stable, and physically consistent models for robot interaction, enabling scalable policy training and evaluation with real-world-like fidelity.

Contribution

The paper presents a novel framework using consistency models for efficient, long-horizon simulation of robot interactions, improving over existing slow and less consistent approaches.

Findings

World models produce interaction-consistent pixel predictions

Policies trained on world-model data perform comparably to real data-trained policies

Strong correlation observed between simulated and real-world policy performance

Abstract

Action-conditioned video prediction models (often referred to as world models) have shown strong potential for robotics applications, but existing approaches are often slow and struggle to capture physically consistent interactions over long horizons, limiting their usefulness for scalable robot policy training and evaluation. We present Interactive World Simulator, a framework for building interactive world models from a moderate-sized robot interaction dataset. Our approach leverages consistency models for both image decoding and latent-space dynamics prediction, enabling fast and stable simulation of physical interactions. In our experiments, the learned world models produce interaction-consistent pixel-level predictions and support stable long-horizon interactions for more than 10 minutes at 15 FPS on a single RTX 4090 GPU. Our framework enables scalable demonstration collection solely within the world models to train state-of-the-art imitation policies. Through extensive real-world evaluation across diverse tasks involving rigid objects, deformable objects, object piles, and their interactions, we find that policies trained on world-model-generated data perform comparably to those trained on the same amount of real-world data. Additionally, we evaluate policies both within the world models and in the real world across diverse tasks, and observe a strong correlation between simulated and real-world performance. Together, these results establish the Interactive World Simulator as a stable and physically consistent surrogate for scalable robotic data generation and faithful, reproducible policy evaluation.