Data Augmented 3D Semantic Scene Completion with 2D Segmentation Priors

TL;DR

This paper introduces SPAwN, a lightweight multimodal 3D CNN that fuses depth data with 2D segmentation priors for improved semantic scene completion, and proposes a novel 3D data augmentation strategy for training.

Contribution

The paper presents SPAwN, a new multimodal 3D CNN architecture and a 3D data augmentation method tailored for semantic scene completion tasks.

Findings

SPAwN outperforms previous methods with similar complexity.

The 3D augmentation strategy improves generalization.

Comprehensive ablation studies validate the approach.

Abstract

Semantic scene completion (SSC) is a challenging Computer Vision task with many practical applications, from robotics to assistive computing. Its goal is to infer the 3D geometry in a field of view of a scene and the semantic labels of voxels, including occluded regions. In this work, we present SPAwN, a novel lightweight multimodal 3D deep CNN that seamlessly fuses structural data from the depth component of RGB-D images with semantic priors from a bimodal 2D segmentation network. A crucial difficulty in this field is the lack of fully labeled real-world 3D datasets which are large enough to train the current data-hungry deep 3D CNNs. In 2D computer vision tasks, many data augmentation strategies have been proposed to improve the generalization ability of CNNs. However those approaches cannot be directly applied to the RGB-D input and output volume of SSC solutions. In this paper, we introduce the use of a 3D data augmentation strategy that can be applied to multimodal SSC networks. We validate our contributions with a comprehensive and reproducible ablation study. Our solution consistently surpasses previous works with a similar level of complexity.