PEGASUS: Physically Enhanced Gaussian Splatting Simulation System for 6DoF Object Pose Dataset Generation

TL;DR

PEGASUS is a versatile simulation system that generates diverse 6DoF object pose datasets using Gaussian Splatting and physics-based scene composition, facilitating effective training and transfer to real-world applications.

Contribution

The paper introduces PEGASUS, a novel physics-enhanced Gaussian Splatting simulation system for creating diverse, realistic 6DoF object pose datasets from commodity camera data.

Findings

Training with PEGASUS data improves pose estimation transfer to real-world data.

PEGASUS enables the creation of extensive static and dynamic scenes for dataset generation.

The Ramen dataset provides a new benchmark with 30 Japanese noodle items for pose estimation.

Abstract

We introduce Physically Enhanced Gaussian Splatting Simulation System (PEGASUS) for 6DOF object pose dataset generation, a versatile dataset generator based on 3D Gaussian Splatting. Environment and object representations can be easily obtained using commodity cameras to reconstruct with Gaussian Splatting. <i>PEGASUS</i> allows the composition of new scenes by merging the respective underlying Gaussian Splatting point cloud of an environment with one or multiple objects. Leveraging a physics engine enables the simulation of natural object placement within a scene through interaction between meshes extracted for the objects and the environment. Consequently, an extensive amount of new scenes - static or dynamic - can be created by combining different environments and objects. By rendering scenes from various perspectives, diverse data points such as RGB images, depth maps, semantic masks, and 6DoF object poses can be extracted. Our study demonstrates that training on data generated by PEGASUS enables pose estimation networks to successfully transfer from synthetic data to real-world data. Moreover, we introduce the Ramen dataset, comprising 30 Japanese cup noodle items. This dataset includes spherical scans that captures images from both object hemisphere and the Gaussian Splatting reconstruction, making them compatible with PEGASUS.