UP-Fuse: Uncertainty-guided LiDAR-Camera Fusion for 3D Panoptic Segmentation

TL;DR

UP-Fuse is a robust uncertainty-aware LiDAR-camera fusion framework for 3D panoptic segmentation that maintains high performance under sensor degradation and failures by dynamically weighting visual cues based on learned uncertainty maps.

Contribution

It introduces a novel uncertainty-guided fusion module and a hybrid 2D-3D transformer, enhancing robustness and accuracy in adverse conditions for 3D perception.

Findings

Outperforms existing methods on Panoptic nuScenes, SemanticKITTI, and Panoptic Waymo benchmarks.

Maintains high segmentation accuracy under severe visual degradation.

Demonstrates robustness to sensor failure and calibration drift.

Abstract

LiDAR-camera fusion enhances 3D panoptic segmentation by leveraging camera images to complement sparse LiDAR scans, but it also introduces a critical failure mode. Under adverse conditions, degradation or failure of the camera sensor can significantly compromise the reliability of the perception system. To address this problem, we introduce UP-Fuse, a novel uncertainty-aware fusion framework in the 2D range-view that remains robust under camera sensor degradation, calibration drift, and sensor failure. Raw LiDAR data is first projected into the range-view and encoded by a LiDAR encoder, while camera features are simultaneously extracted and projected into the same shared space. At its core, UP-Fuse employs an uncertainty-guided fusion module that dynamically modulates cross-modal interaction using predicted uncertainty maps. These maps are learned by quantifying representational divergence under diverse visual degradations, ensuring that only reliable visual cues influence the fused representation. The fused range-view features are decoded by a novel hybrid 2D-3D transformer that mitigates spatial ambiguities inherent to the 2D projection and directly predicts 3D panoptic segmentation masks. Extensive experiments on Panoptic nuScenes, SemanticKITTI, and our introduced Panoptic Waymo benchmark demonstrate the efficacy and robustness of UP-Fuse, which maintains strong performance even under severe visual corruption or misalignment, making it well suited for robotic perception in safety-critical settings.