{\Psi}-Map: Panoptic Surface Integrated Mapping Enables Real2Sim Transfer

TL;DR

This paper introduces -Map, a real-time, large-scale panoptic surface mapping framework that integrates geometric accuracy, coherent understanding, and efficient rendering for robotics perception and simulation.

Contribution

It presents a novel end-to-end learning architecture with LiDAR-based Gaussian models and optimized rendering, enabling high-precision, real-time panoptic mapping in large environments.

Findings

Achieves over 40 FPS inference rate in large-scale scenes.

Demonstrates superior geometric and panoptic reconstruction quality.

Effectively integrates multimodal data for realistic environment modeling.

Abstract

Open-vocabulary panoptic reconstruction is essential for advanced robotics perception and simulation. However, existing methods based on 3D Gaussian Splatting (3DGS) often struggle to simultaneously achieve geometric accuracy, coherent panoptic understanding, and real-time inference frequency in large-scale scenes. In this paper, we propose a comprehensive framework that integrates geometric reinforcement, end-to-end panoptic learning, and efficient rendering. First, to ensure physical realism in large-scale environments, we leverage LiDAR data to construct plane-constrained multimodal Gaussian Mixture Models (GMMs) and employ 2D Gaussian surfels as the map representation, enabling high-precision surface alignment and continuous geometric supervision. Building upon this, to overcome the error accumulation and cumbersome cross-frame association inherent in traditional multi-stage panoptic segmentation pipelines, we design a query-guided end-to-end learning architecture. By utilizing a local cross-attention mechanism within the view frustum, the system lifts 2D mask features directly into 3D space, achieving globally consistent panoptic understanding. Finally, addressing the computational bottlenecks caused by high-dimensional semantic features, we introduce Precise Tile Intersection and a Top-K Hard Selection strategy to optimize the rendering pipeline. Experimental results demonstrate that our system achieves superior geometric and panoptic reconstruction quality in large-scale scenes while maintaining an inference rate exceeding 40 FPS, meeting the real-time requirements of robotic control loops.