RoboPearls: Editable Video Simulation for Robot Manipulation

TL;DR

RoboPearls introduces an editable, photo-realistic video simulation platform for robot manipulation that leverages advanced modules and large language models to improve simulation fidelity and automate production, bridging the sim-to-real gap.

Contribution

The paper presents RoboPearls, a novel simulation framework that constructs view-consistent, editable robotic manipulation videos from demonstration data using 3D Gaussian Splatting and LLM integration.

Findings

Effective simulation across multiple datasets and scenes.

Demonstrated improvement in sim-to-real transfer.

Automated simulation production via LLMs.

Abstract

The development of generalist robot manipulation policies has seen significant progress, driven by large-scale demonstration data across diverse environments. However, the high cost and inefficiency of collecting real-world demonstrations hinder the scalability of data acquisition. While existing simulation platforms enable controlled environments for robotic learning, the challenge of bridging the sim-to-real gap remains. To address these challenges, we propose RoboPearls, an editable video simulation framework for robotic manipulation. Built on 3D Gaussian Splatting (3DGS), RoboPearls enables the construction of photo-realistic, view-consistent simulations from demonstration videos, and supports a wide range of simulation operators, including various object manipulations, powered by advanced modules like Incremental Semantic Distillation (ISD) and 3D regularized NNFM Loss (3D-NNFM). Moreover, by incorporating large language models (LLMs), RoboPearls automates the simulation production process in a user-friendly manner through flexible command interpretation and execution. Furthermore, RoboPearls employs a vision-language model (VLM) to analyze robotic learning issues to close the simulation loop for performance enhancement. To demonstrate the effectiveness of RoboPearls, we conduct extensive experiments on multiple datasets and scenes, including RLBench, COLOSSEUM, Ego4D, Open X-Embodiment, and a real-world robot, which demonstrate our satisfactory simulation performance.