PanoSLAM: Panoptic 3D Scene Reconstruction via Gaussian SLAM

TL;DR

PanoSLAM is a novel SLAM system that integrates geometric, semantic, and instance segmentation into a unified 3D scene reconstruction framework using Gaussian Splatting and an online Spatial-Temporal Lifting module.

Contribution

It introduces the first panoptic 3D scene reconstruction system from RGB-D videos, combining multiple segmentation tasks within a Gaussian Splatting-based SLAM framework.

Findings

Outperforms recent semantic SLAM methods in mapping and tracking accuracy

Achieves panoptic 3D reconstruction of open-world environments from RGB-D videos

Effectively refines 2D predictions into coherent 3D representations

Abstract

Understanding geometric, semantic, and instance information in 3D scenes from sequential video data is essential for applications in robotics and augmented reality. However, existing Simultaneous Localization and Mapping (SLAM) methods generally focus on either geometric or semantic reconstruction. In this paper, we introduce PanoSLAM, the first SLAM system to integrate geometric reconstruction, 3D semantic segmentation, and 3D instance segmentation within a unified framework. Our approach builds upon 3D Gaussian Splatting, modified with several critical components to enable efficient rendering of depth, color, semantic, and instance information from arbitrary viewpoints. To achieve panoptic 3D scene reconstruction from sequential RGB-D videos, we propose an online Spatial-Temporal Lifting (STL) module that transfers 2D panoptic predictions from vision models into 3D Gaussian representations. This STL module addresses the challenges of label noise and inconsistencies in 2D predictions by refining the pseudo labels across multi-view inputs, creating a coherent 3D representation that enhances segmentation accuracy. Our experiments show that PanoSLAM outperforms recent semantic SLAM methods in both mapping and tracking accuracy. For the first time, it achieves panoptic 3D reconstruction of open-world environments directly from the RGB-D video. (https://github.com/runnanchen/PanoSLAM)