Point Cloud Scene Completion with Joint Color and Semantic Estimation from Single RGB-D Image

TL;DR

This paper introduces a deep reinforcement learning approach for scene completion from a single RGB-D image, integrating semantic segmentation, volumetric reconstruction, and multi-view inpainting to achieve high-quality 3D scene reconstruction.

Contribution

It presents a novel end-to-end method combining view inpainting and reinforcement learning for scene completion, outperforming existing methods on 3D-FUTURE data.

Findings

Achieves superior qualitative and quantitative reconstruction results.

Effectively completes occluded scene regions with multi-view inpainting.

Outperforms state-of-the-art methods on benchmark datasets.

Abstract

We present a deep reinforcement learning method of progressive view inpainting for colored semantic point cloud scene completion under volume guidance, achieving high-quality scene reconstruction from only a single RGB-D image with severe occlusion. Our approach is end-to-end, consisting of three modules: 3D scene volume reconstruction, 2D RGB-D and segmentation image inpainting, and multi-view selection for completion. Given a single RGB-D image, our method first predicts its semantic segmentation map and goes through the 3D volume branch to obtain a volumetric scene reconstruction as a guide to the next view inpainting step, which attempts to make up the missing information; the third step involves projecting the volume under the same view of the input, concatenating them to complete the current view RGB-D and segmentation map, and integrating all RGB-D and segmentation maps into the point cloud. Since the occluded areas are unavailable, we resort to a A3C network to glance around and pick the next best view for large hole completion progressively until a scene is adequately reconstructed while guaranteeing validity. All steps are learned jointly to achieve robust and consistent results. We perform qualitative and quantitative evaluations with extensive experiments on the 3D-FUTURE data, obtaining better results than state-of-the-arts.