OccDepth: A Depth-Aware Method for 3D Semantic Scene Completion

TL;DR

OccDepth introduces a stereo-based 3D semantic scene completion method that leverages implicit depth information and novel modules to improve geometric and semantic reconstruction accuracy in autonomous systems.

Contribution

The paper presents the first stereo SSC method, OccDepth, with modules for better depth-aware feature fusion and geometry-aware 3D feature extraction, advancing 3D scene understanding.

Findings

Achieves +4.82% mIoU improvement over state-of-the-art methods.

Stereo images contribute +2.49% mIoU, depth-aware modules contribute +2.33%.

Demonstrates superior performance on SemanticKITTI dataset.

Abstract

3D Semantic Scene Completion (SSC) can provide dense geometric and semantic scene representations, which can be applied in the field of autonomous driving and robotic systems. It is challenging to estimate the complete geometry and semantics of a scene solely from visual images, and accurate depth information is crucial for restoring 3D geometry. In this paper, we propose the first stereo SSC method named OccDepth, which fully exploits implicit depth information from stereo images (or RGBD images) to help the recovery of 3D geometric structures. The Stereo Soft Feature Assignment (Stereo-SFA) module is proposed to better fuse 3D depth-aware features by implicitly learning the correlation between stereo images. In particular, when the input are RGBD image, a virtual stereo images can be generated through original RGB image and depth map. Besides, the Occupancy Aware Depth (OAD) module is used to obtain geometry-aware 3D features by knowledge distillation using pre-trained depth models. In addition, a reformed TartanAir benchmark, named SemanticTartanAir, is provided in this paper for further testing our OccDepth method on SSC task. Compared with the state-of-the-art RGB-inferred SSC method, extensive experiments on SemanticKITTI show that our OccDepth method achieves superior performance with improving +4.82% mIoU, of which +2.49% mIoU comes from stereo images and +2.33% mIoU comes from our proposed depth-aware method. Our code and trained models are available at https://github.com/megvii-research/OccDepth.