3D-to-2D Distillation for Indoor Scene Parsing

TL;DR

This paper introduces a 3D-to-2D distillation framework that leverages 3D data to improve indoor scene parsing from RGB images, enabling better feature learning without requiring 3D data at inference.

Contribution

The work presents a novel distillation method, a two-stage normalization scheme, and a semantic-aware adversarial training approach for enhanced 2D scene parsing using 3D knowledge.

Findings

Improved accuracy on ScanNet-V2, S3DIS, and NYU-v2 datasets.

Enhanced model generalization to unseen data.

Effective training with unpaired 3D data.

Abstract

Indoor scene semantic parsing from RGB images is very challenging due to occlusions, object distortion, and viewpoint variations. Going beyond prior works that leverage geometry information, typically paired depth maps, we present a new approach, a 3D-to-2D distillation framework, that enables us to leverage 3D features extracted from large-scale 3D data repository (e.g., ScanNet-v2) to enhance 2D features extracted from RGB images. Our work has three novel contributions. First, we distill 3D knowledge from a pretrained 3D network to supervise a 2D network to learn simulated 3D features from 2D features during the training, so the 2D network can infer without requiring 3D data. Second, we design a two-stage dimension normalization scheme to calibrate the 2D and 3D features for better integration. Third, we design a semantic-aware adversarial training model to extend our framework for training with unpaired 3D data. Extensive experiments on various datasets, ScanNet-V2, S3DIS, and NYU-v2, demonstrate the superiority of our approach. Also, experimental results show that our 3D-to-2D distillation improves the model generalization.