OnlinePG: Online Open-Vocabulary Panoptic Mapping with 3D Gaussian Splatting

TL;DR

OnlinePG introduces an online, open-vocabulary panoptic mapping system that combines geometric reconstruction and semantic understanding using 3D Gaussian Splatting, enabling real-time scene perception for robotics.

Contribution

It is the first online system integrating 3D Gaussian Splatting with open-vocabulary perception for panoptic mapping in real-time.

Findings

Outperforms existing online methods in accuracy.

Maintains real-time processing speed.

Effectively fuses geometric and semantic cues.

Abstract

Open-vocabulary scene understanding with online panoptic mapping is essential for embodied applications to perceive and interact with environments. However, existing methods are predominantly offline or lack instance-level understanding, limiting their applicability to real-world robotic tasks. In this paper, we propose OnlinePG, a novel and effective system that integrates geometric reconstruction and open-vocabulary perception using 3D Gaussian Splatting in an online setting. Technically, to achieve online panoptic mapping, we employ an efficient local-to-global paradigm with a sliding window. To build local consistency map, we construct a 3D segment clustering graph that jointly leverages geometric and semantic cues, fusing inconsistent segments within sliding window into complete instances. Subsequently, to update the global map, we construct explicit grids with spatial attributes for the local 3D Gaussian map and fuse them into the global map via robust bidirectional bipartite 3D Gaussian instance matching. Finally, we utilize the fused VLM features inside the 3D spatial attribute grids to achieve open-vocabulary scene understanding. Extensive experiments on widely used datasets demonstrate that our method achieves better performance among online approaches, while maintaining real-time efficiency.