Bridging Simulation and Reality: Cross-Domain Transfer with Semantic 2D Gaussian Splatting

TL;DR

This paper introduces Semantic 2D Gaussian Splatting (S2GS), a novel method for extracting domain-invariant features to improve the transfer of robotic manipulation policies from simulation to real-world environments.

Contribution

The paper proposes S2GS, a new representation that constructs multi-view semantic fields and projects them into 3D space, enhancing sim-to-real transferability in robotic manipulation.

Findings

S2GS significantly improves sim-to-real transfer performance.

S2GS maintains high task success rates in real-world deployment.

Semantic filtering ensures clean, consistent inputs for policy learning.

Abstract

Cross-domain transfer in robotic manipulation remains a longstanding challenge due to the significant domain gap between simulated and real-world environments. Existing methods such as domain randomization, adaptation, and sim-real calibration often require extensive tuning or fail to generalize to unseen scenarios. To address this issue, we observe that if domain-invariant features are utilized during policy training in simulation, and the same features can be extracted and provided as the input to policy during real-world deployment, the domain gap can be effectively bridged, leading to significantly improved policy generalization. Accordingly, we propose Semantic 2D Gaussian Splatting (S2GS), a novel representation method that extracts object-centric, domain-invariant spatial features. S2GS constructs multi-view 2D semantic fields and projects them into a unified 3D space via feature-level Gaussian splatting. A semantic filtering mechanism removes irrelevant background content, ensuring clean and consistent inputs for policy learning. To evaluate the effectiveness of S2GS, we adopt Diffusion Policy as the downstream learning algorithm and conduct experiments in the ManiSkill simulation environment, followed by real-world deployment. Results demonstrate that S2GS significantly improves sim-to-real transferability, maintaining high and stable task performance in real-world scenarios.