RoboLab: A High-Fidelity Simulation Benchmark for Analysis of Task Generalist Policies

TL;DR

RoboLab is a high-fidelity simulation benchmark designed to evaluate the true generalization of task-generalist robotic policies through diverse tasks and systematic analysis of policy robustness.

Contribution

It introduces RoboLab, a scalable simulation framework with 120 tasks and a systematic analysis method to assess policy performance and robustness.

Findings

Current state-of-the-art models show significant performance gaps in RoboLab.

RoboLab enables analysis of policy sensitivity to controlled perturbations.

The benchmark provides granular metrics for evaluating generalization.

Abstract

The pursuit of general-purpose robotics has yielded impressive foundation models, yet simulation-based benchmarking remains a bottleneck due to rapid performance saturation and a lack of true generalization testing. Existing benchmarks often exhibit significant domain overlap between training and evaluation, trivializing success rates and obscuring insights into robustness. We introduce RoboLab, a simulation benchmarking framework designed to address these challenges. Concretely, our framework is designed to answer two questions: (1) to what extent can we understand the performance of a real-world policy by analyzing its behavior in simulation, and (2) which factor most strongly affect policy behavior. First, RoboLab enables human-authored and LLM-enabled generation of scenes and tasks in a robot- and policy-agnostic manner within a high-fidelity simulation environment. We introduce an accompanying RoboLab-120 benchmark, consisting of 120 tasks categorized into three competency axes: visual, procedural, relational, across three difficulty levels. Second, we introduce a systematic analysis of real-world policies that quantify both their performance and the sensitivity of their behavior to controlled perturbations, exposing significant performance gap in current state-of-the-art models. By providing granular metrics and a scalable toolset, RoboLab offers a scalable framework for evaluating the true generalization capabilities of task-generalist robotic policies. Project website: https://research.nvidia.com/labs/srl/projects/robolab/.