3DMV: Joint 3D-Multi-View Prediction for 3D Semantic Scene Segmentation

TL;DR

3DMV introduces a joint 3D-multi-view prediction network that combines RGB and geometric data for improved indoor 3D semantic scene segmentation, significantly outperforming existing methods on benchmark datasets.

Contribution

The paper presents a novel end-to-end architecture that fuses RGB image features with 3D geometry using differentiable backprojection and multi-view pooling.

Findings

Achieves 75% accuracy on ScanNet benchmark, surpassing previous methods.

Effectively combines multi-view RGB features with 3D data for better segmentation.

Outperforms existing volumetric architectures in 3D scene understanding.

Abstract

We present 3DMV, a novel method for 3D semantic scene segmentation of RGB-D scans in indoor environments using a joint 3D-multi-view prediction network. In contrast to existing methods that either use geometry or RGB data as input for this task, we combine both data modalities in a joint, end-to-end network architecture. Rather than simply projecting color data into a volumetric grid and operating solely in 3D -- which would result in insufficient detail -- we first extract feature maps from associated RGB images. These features are then mapped into the volumetric feature grid of a 3D network using a differentiable backprojection layer. Since our target is 3D scanning scenarios with possibly many frames, we use a multi-view pooling approach in order to handle a varying number of RGB input views. This learned combination of RGB and geometric features with our joint 2D-3D architecture achieves significantly better results than existing baselines. For instance, our final result on the ScanNet 3D segmentation benchmark increases from 52.8\% to 75\% accuracy compared to existing volumetric architectures.