RoboGSim: A Real2Sim2Real Robotic Gaussian Splatting Simulator

TL;DR

RoboGSim is a high-fidelity, physics-aware robotic simulator using Gaussian Splatting that enables efficient large-scale data collection and improves sim2real transfer for manipulation policies.

Contribution

It introduces RoboGSim, a novel real2sim2real robotic simulator leveraging 3D Gaussian Splatting and physics engine for high-fidelity, scalable data synthesis and evaluation.

Findings

RoboGSim achieves high consistency with real robot data in texture and physics.

Zero-shot transfer performance on real robots is achieved with RoboGSim data.

RoboGSim outperforms real robot data in novel perspectives and scenes.

Abstract

Efficient acquisition of real-world embodied data has been increasingly critical. However, large-scale demonstrations captured by remote operation tend to take extremely high costs and fail to scale up the data size in an efficient manner. Sampling the episodes under a simulated environment is a promising way for large-scale collection while existing simulators fail to high-fidelity modeling on texture and physics. To address these limitations, we introduce the RoboGSim, a real2sim2real robotic simulator, powered by 3D Gaussian Splatting and the physics engine. RoboGSim mainly includes four parts: Gaussian Reconstructor, Digital Twins Builder, Scene Composer, and Interactive Engine. It can synthesize the simulated data with novel views, objects, trajectories, and scenes. RoboGSim also provides an online, reproducible, and safe evaluation for different manipulation policies. The real2sim and sim2real transfer experiments show a high consistency in the texture and physics. We compared the test results of RoboGSim data and real robot data on both RoboGSim and real robot platforms. The experimental results show that the RoboGSim data model can achieve zero-shot performance on the real robot, with results comparable to real robot data. Additionally, in experiments with novel perspectives and novel scenes, the RoboGSim data model performed even better on the real robot than the real robot data model. This not only helps reduce the sim2real gap but also addresses the limitations of real robot data collection, such as its single-source and high cost. We hope RoboGSim serves as a closed-loop simulator for fair comparison on policy learning. More information can be found on our project page https://robogsim.github.io/.