Unleashing Semantic and Geometric Priors for 3D Scene Completion

TL;DR

FoundationSSC introduces a dual-decoupling framework with a foundation encoder and Axis-Aware Fusion for improved 3D scene completion, significantly enhancing semantic and geometric accuracy in autonomous navigation.

Contribution

The paper presents a novel dual-decoupling approach with specialized pathways and a fusion module, enabling better disentanglement and integration of semantic and geometric priors.

Findings

Achieves +0.23 mIoU and +2.03 IoU improvements on SemanticKITTI.

Sets new state-of-the-art on SSCBench-KITTI-360 with 21.78 mIoU.

Demonstrates superior performance in both semantic and geometric metrics.

Abstract

Camera-based 3D semantic scene completion (SSC) provides dense geometric and semantic perception for autonomous driving and robotic navigation. However, existing methods rely on a coupled encoder to deliver both semantic and geometric priors, which forces the model to make a trade-off between conflicting demands and limits its overall performance. To tackle these challenges, we propose FoundationSSC, a novel framework that performs dual decoupling at both the source and pathway levels. At the source level, we introduce a foundation encoder that provides rich semantic feature priors for the semantic branch and high-fidelity stereo cost volumes for the geometric branch. At the pathway level, these priors are refined through specialised, decoupled pathways, yielding superior semantic context and depth distributions. Our dual-decoupling design produces disentangled and refined inputs, which are then utilised by a hybrid view transformation to generate complementary 3D features. Additionally, we introduce a novel Axis-Aware Fusion (AAF) module that addresses the often-overlooked challenge of fusing these features by anisotropically merging them into a unified representation. Extensive experiments demonstrate the advantages of FoundationSSC, achieving simultaneous improvements in both semantic and geometric metrics, surpassing prior bests by +0.23 mIoU and +2.03 IoU on SemanticKITTI. Additionally, we achieve state-of-the-art performance on SSCBench-KITTI-360, with 21.78 mIoU and 48.61 IoU.