TACOcc:Target-Adaptive Cross-Modal Fusion with Volume Rendering for 3D Semantic Occupancy

TL;DR

TACOcc introduces an adaptive cross-modal fusion method with volume rendering supervision to improve 3D semantic occupancy prediction, addressing geometry-semantics mismatch and surface detail loss in multi-modal data.

Contribution

The paper proposes a novel target-scale adaptive bidirectional retrieval mechanism and an improved volume rendering pipeline for enhanced multi-modal 3D occupancy prediction.

Findings

Outperforms existing methods on nuScenes and SemanticKITTI benchmarks.

Effectively aligns features across modalities with adaptive neighborhood expansion and shrinking.

Enhances surface detail reconstruction via volume rendering supervision.

Abstract

The performance of multi-modal 3D occupancy prediction is limited by ineffective fusion, mainly due to geometry-semantics mismatch from fixed fusion strategies and surface detail loss caused by sparse, noisy annotations. The mismatch stems from the heterogeneous scale and distribution of point cloud and image features, leading to biased matching under fixed neighborhood fusion. To address this, we propose a target-scale adaptive, bidirectional symmetric retrieval mechanism. It expands the neighborhood for large targets to enhance context awareness and shrinks it for small ones to improve efficiency and suppress noise, enabling accurate cross-modal feature alignment. This mechanism explicitly establishes spatial correspondences and improves fusion accuracy. For surface detail loss, sparse labels provide limited supervision, resulting in poor predictions for small objects. We introduce an improved volume rendering pipeline based on 3D Gaussian Splatting, which takes fused features as input to render images, applies photometric consistency supervision, and jointly optimizes 2D-3D consistency. This enhances surface detail reconstruction while suppressing noise propagation. In summary, we propose TACOcc, an adaptive multi-modal fusion framework for 3D semantic occupancy prediction, enhanced by volume rendering supervision. Experiments on the nuScenes and SemanticKITTI benchmarks validate its effectiveness.