Robotic Learning in your Backyard: A Neural Simulator from Open Source Components

TL;DR

SplatGym is an open-source neural simulator that creates photorealistic virtual environments from videos, enabling reinforcement learning for robotic control without proprietary tools.

Contribution

It introduces SplatGym, the first open-source neural simulator that generates realistic environments from videos for robotic learning applications.

Findings

Successfully trained visual navigation policies using reinforcement learning.

Demonstrated the creation of photorealistic environments from a single video.

Enabled collision detection and virtual object in-painting in the simulator.

Abstract

The emergence of 3D Gaussian Splatting for fast and high-quality novel view synthesize has opened up the possibility to construct photo-realistic simulations from video for robotic reinforcement learning. While the approach has been demonstrated in several research papers, the software tools used to build such a simulator remain unavailable or proprietary. We present SplatGym, an open source neural simulator for training data-driven robotic control policies. The simulator creates a photorealistic virtual environment from a single video. It supports ego camera view generation, collision detection, and virtual object in-painting. We demonstrate training several visual navigation policies via reinforcement learning. SplatGym represents a notable first step towards an open-source general-purpose neural environment for robotic learning. It broadens the range of applications that can effectively utilise reinforcement learning by providing convenient and unrestricted tooling, and by eliminating the need for the manual development of conventional 3D environments.