ManipArena: Comprehensive Real-world Evaluation of Reasoning-Oriented Generalist Robot Manipulation

TL;DR

ManipArena is a comprehensive evaluation framework that bridges simulation and real-world testing for reasoning-oriented robot manipulation, addressing current limitations in benchmarks and enabling fair comparison of models.

Contribution

It introduces a standardized, realistic, and scalable evaluation platform with diverse tasks, diagnostics, and real-to-sim environments for assessing VLA and world models in robotics.

Findings

Supports 20 diverse tasks with 10,812 expert trajectories.

Enables evaluation of reasoning, generalization, and long-horizon manipulation.

Provides rich sensory diagnostics and real-to-sim environments.

Abstract

Vision-Language-Action (VLA) models and world models have recently emerged as promising paradigms for general-purpose robotic intelligence, yet their progress is hindered by the lack of reliable evaluation protocols that reflect real-world deployment. Existing benchmarks are largely simulator-centric, which provide controllability but fail to capture the reality gap caused by perception noise, complex contact dynamics, hardware constraints, and system latency. Moreover, fragmented real-world evaluations across different robot platforms prevent fair and reproducible comparison. To address these challenges, we introduce ManipArena, a standardized evaluation framework designed to bridge simulation and real-world execution. ManipArena comprises 20 diverse tasks across 10,812 expert trajectories emphasizing reasoning-oriented manipulation tasks requiring semantic and spatial reasoning, supports multi-level generalization through controlled out-of-distribution settings, and incorporates long-horizon mobile manipulation beyond tabletop scenarios. The framework further provides rich sensory diagnostics, including low-level motor signals, and synchronized real-to-sim environments constructed via high-quality 3D scanning. Together, these features enable fair, realistic, and reproducible evaluation for both VLA and world model approaches, providing a scalable foundation for diagnosing and advancing embodied intelligence systems.