dWorldEval: Scalable Robotic Policy Evaluation via Discrete Diffusion World Model

TL;DR

dWorldEval introduces a scalable, transformer-based discrete diffusion world model that unifies multiple modalities for efficient robotics policy evaluation across diverse tasks and environments.

Contribution

The paper presents dWorldEval, a novel approach using a discrete diffusion world model with a transformer network and memory mechanisms for scalable robotics policy evaluation.

Findings

dWorldEval outperforms previous methods like WorldEval, Ctrl-World, and WorldGym.

It effectively models vision, language, and actions in a unified token space.

The approach enables evaluation across thousands of environments and tasks.

Abstract

Evaluating robotics policies across thousands of environments and thousands of tasks is infeasible with existing approaches. This motivates the need for a new methodology for scalable robotics policy evaluation. In this paper, we propose dWorldEval, which uses a discrete diffusion world model as a scalable evaluation proxy for robotics policies. Specifically, dWorldEval maps all modalities - including vision, language, and robotic actions - into a unified token space, modeling them via a single transformer-based denoising network. In this paper, we propose dWorldEval, using a discrete diffusion world model as a scalable evaluation proxy for robotics policy. Specifically, it maps all modalities, including vision, language, and robotics action into a unified token space, then denoises them with a single transformer network. Building on this architecture, we employ a sparse keyframe memory to maintain spatiotemporal consistency. We also introduce a progress token that indicates the degree of task completion. At inference, the model jointly predicts future observations and progress token, allowing automatically determine success when the progress reaches 1. Extensive experiments demonstrate that dWorldEval significantly outperforms previous approaches, i.e., WorldEval, Ctrl-World, and WorldGym, on LIBERO, RoboTwin, and multiple real-robot tasks. It paves the way for a new architectural paradigm in building world simulators for robotics evaluation at scale.