Neural Part Priors: Learning to Optimize Part-Based Object Completion in RGB-D Scans

TL;DR

This paper introduces Neural Part Priors, a method leveraging synthetic 3D shape datasets to improve part-based object completion and decomposition in real-world RGB-D scans, enhancing scene understanding.

Contribution

The paper presents Neural Part Priors, a novel approach for learning geometric part priors from synthetic data and optimizing them for accurate real-world scene object decomposition.

Findings

Outperforms state-of-the-art in part decomposition

Achieves robust object completion in real-world scenes

Enables scene-consistent part segmentation

Abstract

3D object recognition has seen significant advances in recent years, showing impressive performance on real-world 3D scan benchmarks, but lacking in object part reasoning, which is fundamental to higher-level scene understanding such as inter-object similarities or object functionality. Thus, we propose to leverage large-scale synthetic datasets of 3D shapes annotated with part information to learn Neural Part Priors (NPPs), optimizable spaces characterizing geometric part priors. Crucially, we can optimize over the learned part priors in order to fit to real-world scanned 3D scenes at test time, enabling robust part decomposition of the real objects in these scenes that also estimates the complete geometry of the object while fitting accurately to the observed real geometry. Moreover, this enables global optimization over geometrically similar detected objects in a scene, which often share strong geometric commonalities, enabling scene-consistent part decompositions. Experiments on the ScanNet dataset demonstrate that NPPs significantly outperforms state of the art in part decomposition and object completion in real-world scenes.