GSWorld: Closed-Loop Photo-Realistic Simulation Suite for Robotic Manipulation

TL;DR

GSWorld is a photo-realistic simulation platform combining Gaussian Splatting and physics engines, enabling reproducible evaluation and sim2real policy training for robotic manipulation without real robots.

Contribution

Introduction of GSWorld, a novel simulation suite with a new asset format (GSDF) for diverse, photo-realistic robotic manipulation scenarios and applications.

Findings

Effective zero-shot sim2real pixel-to-action policy learning.

Automated high-quality data collection for policy adaptation.

Reproducible benchmarking of manipulation policies.

Abstract

This paper presents GSWorld, a robust, photo-realistic simulator for robotics manipulation that combines 3D Gaussian Splatting with physics engines. Our framework advocates "closing the loop" of developing manipulation policies with reproducible evaluation of policies learned from real-robot data and sim2real policy training without using real robots. To enable photo-realistic rendering of diverse scenes, we propose a new asset format, which we term GSDF (Gaussian Scene Description File), that infuses Gaussian-on-Mesh representation with robot URDF and other objects. With a streamlined reconstruction pipeline, we curate a database of GSDF that contains 3 robot embodiments for single-arm and bimanual manipulation, as well as more than 40 objects. Combining GSDF with physics engines, we demonstrate several immediate interesting applications: (1) learning zero-shot sim2real pixel-to-action manipulation policy with photo-realistic rendering, (2) automated high-quality DAgger data collection for adapting policies to deployment environments, (3) reproducible benchmarking of real-robot manipulation policies in simulation, (4) simulation data collection by virtual teleoperation, and (5) zero-shot sim2real visual reinforcement learning. Website: https://3dgsworld.github.io/.