RoboTwin: Dual-Arm Robot Benchmark with Generative Digital Twins (early version)

TL;DR

RoboTwin introduces a digital twin framework utilizing generative models and large language models to produce diverse datasets and evaluation benchmarks for dual-arm robotic manipulation, improving policy success rates.

Contribution

It presents a novel generative digital twin framework that creates diverse, realistic datasets and benchmarks for dual-arm robotics, bridging the gap between simulation and real-world performance.

Findings

Pre-trained policies on RoboTwin data outperform real-world only training.

Over 70% success rate for single-arm tasks with RoboTwin data.

Over 40% success rate for dual-arm tasks with RoboTwin data.

Abstract

In the rapidly advancing field of robotics, dual-arm coordination and complex object manipulation are essential capabilities for developing advanced autonomous systems. However, the scarcity of diverse, high-quality demonstration data and real-world-aligned evaluation benchmarks severely limits such development. To address this, we introduce RoboTwin, a generative digital twin framework that uses 3D generative foundation models and large language models to produce diverse expert datasets and provide a real-world-aligned evaluation platform for dual-arm robotic tasks. Specifically, RoboTwin creates varied digital twins of objects from single 2D images, generating realistic and interactive scenarios. It also introduces a spatial relation-aware code generation framework that combines object annotations with large language models to break down tasks, determine spatial constraints, and generate precise robotic movement code. Our framework offers a comprehensive benchmark with both simulated and real-world data, enabling standardized evaluation and better alignment between simulated training and real-world performance. We validated our approach using the open-source COBOT Magic Robot platform. Policies pre-trained on RoboTwin-generated data and fine-tuned with limited real-world samples improve the success rate of over 70% for single-arm tasks and over 40% for dual-arm tasks compared to models trained solely on real-world data. This significant improvement demonstrates RoboTwin's potential to enhance the development and evaluation of dual-arm robotic manipulation systems. Project Page: https://robotwin-benchmark.github.io/early-version/.