GS-Playground: A High-Throughput Photorealistic Simulator for Vision-Informed Robot Learning

TL;DR

GS-Playground is a high-throughput, photorealistic simulation framework that accelerates vision-informed robot learning by combining a novel parallel physics engine with efficient scene reconstruction.

Contribution

It introduces a novel high-performance physics engine integrated with a 3D Gaussian Splatting rendering pipeline and an automated workflow for creating photorealistic environments.

Findings

Achieves 10^4 FPS at 640x480 resolution for large-scale visual RL.

Effectively bridges perceptual and physical gaps in diverse embodied tasks.

Streamlines scene creation with automated photorealistic environment reconstruction.

Abstract

Embodied AI research is undergoing a shift toward vision-centric perceptual paradigms. While massively parallel simulators have catalyzed breakthroughs in proprioception-based locomotion, their potential remains largely untapped for vision-informed tasks due to the prohibitive computational overhead of large-scale photorealistic rendering. Furthermore, the creation of simulation-ready 3D assets heavily relies on labor-intensive manual modeling, while the significant sim-to-real physical gap hinders the transfer of contact-rich manipulation policies. To address these bottlenecks, we propose GS-Playground, a multi-modal simulation framework designed to accelerate end-to-end perceptual learning. We develop a novel high-performance parallel physics engine, specifically designed to integrate with a batch 3D Gaussian Splatting (3DGS) rendering pipeline to ensure high-fidelity synchronization. Our system achieves a breakthrough throughput of 10^4 FPS at 640x480 resolution, significantly lowering the barrier for large-scale visual RL. Additionally, we introduce an automated Real2Sim workflow that reconstructs photorealistic, physically consistent, and memory-efficient environments, streamlining the generation of complex simulation-ready scenes. Extensive experiments on locomotion, navigation, and manipulation demonstrate that GS-Playground effectively bridges the perceptual and physical gaps across diverse embodied tasks. Project homepage: https://gsplayground.github.io.