RL-GSBridge: 3D Gaussian Splatting Based Real2Sim2Real Method for Robotic Manipulation Learning

TL;DR

RL-GSBridge leverages 3D Gaussian splatting to enable zero-shot sim-to-real transfer in robotic manipulation, improving realism and success rates without extensive data or large models.

Contribution

The paper introduces RL-GSBridge, a novel framework integrating 3D Gaussian splatting into RL simulation for realistic, zero-shot sim-to-real transfer in vision-based robotics.

Findings

Achieves high success rates in real-world grasping and pick-and-place tasks.

Reduces rendering artifacts in unstructured objects.

Enhances visual realism with mesh-based 3D Gaussian splatting.

Abstract

Sim-to-Real refers to the process of transferring policies learned in simulation to the real world, which is crucial for achieving practical robotics applications. However, recent Sim2real methods either rely on a large amount of augmented data or large learning models, which is inefficient for specific tasks. In recent years, with the emergence of radiance field reconstruction methods, especially 3D Gaussian splatting, it has become possible to construct realistic real-world scenes. To this end, we propose RL-GSBridge, a novel real-to-sim-to-real framework which incorporates 3D Gaussian Splatting into the conventional RL simulation pipeline, enabling zero-shot sim-to-real transfer for vision-based deep reinforcement learning. We introduce a mesh-based 3D GS method with soft binding constraints, enhancing the rendering quality of mesh models. Then utilizing a GS editing approach to synchronize the rendering with the physics simulator, RL-GSBridge could reflect the visual interactions of the physical robot accurately. Through a series of sim-to-real experiments, including grasping and pick-and-place tasks, we demonstrate that RL-GSBridge maintains a satisfactory success rate in real-world task completion during sim-to-real transfer. Furthermore, a series of rendering metrics and visualization results indicate that our proposed mesh-based 3D GS reduces artifacts in unstructured objects, demonstrating more realistic rendering performance.