RobotArena $\infty$: Scalable Robot Benchmarking via Real-to-Sim Translation

TL;DR

RobotArena Infinity offers a scalable, reproducible benchmarking framework for robot policies by translating real-world demonstrations into simulated environments with human feedback, enabling large-scale evaluation of generalist robots.

Contribution

It introduces a novel simulation-based benchmarking approach that leverages vision-language models and human preferences to evaluate robot policies at scale.

Findings

Automated conversion of video demonstrations into simulated environments.

Scalable human preference collection for policy evaluation.

Robustness testing through environment perturbations.

Abstract

The pursuit of robot generalists, agents capable of performing diverse tasks across diverse environments, demands rigorous and scalable evaluation. Yet real-world testing of robot policies remains fundamentally constrained: it is labor-intensive, slow, unsafe at scale, and difficult to reproduce. As policies expand in scope and complexity, these barriers only intensify, since defining "success" in robotics often hinges on nuanced human judgments of execution quality. We introduce RobotArena Infinity, a new benchmarking framework that overcomes these challenges by shifting vision-language-action (VLA) evaluation into large-scale simulated environments augmented with online human feedback. Leveraging advances in vision-language models, 2D-to-3D generative modeling, and differentiable rendering, our approach automatically converts video demonstrations from widely used robot datasets into simulated counterparts. Within these digital twins, we assess VLA policies using both automated vision-language-model-guided scoring and scalable human preference judgments collected from crowdworkers, transforming human involvement from tedious scene setup, resetting, and safety supervision into lightweight preference comparisons. To measure robustness, we systematically perturb simulated environments along multiple axes, including textures and object placements, stress-testing policy generalization under controlled variation. The result is a continuously evolving, reproducible, and scalable benchmark for real-world-trained robot manipulation policies, addressing a critical missing capability in today's robotics landscape.