DSOcc: Leveraging Depth Awareness and Semantic Aid to Boost Camera-Based 3D Semantic Occupancy Prediction

TL;DR

DSOcc introduces a novel approach combining depth awareness and semantic aid to improve camera-based 3D semantic occupancy prediction, achieving state-of-the-art results by jointly inferring occupancy states and classes with robust multi-frame fusion.

Contribution

The paper proposes DSOcc, a method that jointly infers occupancy states and classes using depth and semantic cues, enhancing accuracy and robustness over previous approaches.

Findings

Achieves state-of-the-art performance on SemanticKITTI dataset.

Demonstrates robustness through multi-frame semantic and occupancy fusion.

Outperforms existing camera-based 3D occupancy prediction methods.

Abstract

Camera-based 3D semantic occupancy prediction offers an efficient and cost-effective solution for perceiving surrounding scenes in autonomous driving. However, existing works rely on explicit occupancy state inference, leading to numerous incorrect feature assignments, and insufficient samples restrict the learning of occupancy class inference. To address these challenges, we propose leveraging \textbf{D}epth awareness and \textbf{S}emantic aid to boost camera-based 3D semantic \textbf{Occ}upancy prediction (\textbf{DSOcc}). We jointly perform occupancy state and occupancy class inference, where soft occupancy confidence is calculated by non-learning method and multiplied with image features to make voxels aware of depth, enabling adaptive implicit occupancy state inference. Instead of enhancing feature learning, we directly utilize well-trained image semantic segmentation and fuse multiple frames with their occupancy probabilities to aid occupancy class inference, thereby enhancing robustness. Experimental results demonstrate that DSOcc achieves state-of-the-art performance on the SemanticKITTI dataset among camera-based methods and achieves competitive performance on the SSCBench-KITTI-360 and Occ3D-nuScenes datasets. Code will be released on github.