The Neurally-Guided Shape Parser: Grammar-based Labeling of 3D Shape Regions with Approximate Inference

TL;DR

The paper introduces NGSP, a neural-guided probabilistic approach for fine-grained semantic labeling of 3D shape regions, improving accuracy and robustness over existing methods, and applicable to CAD models with limited data.

Contribution

NGSP is a novel neural-guided inference method that enhances semantic segmentation of 3D shapes by modeling the posterior distribution with approximate MAP inference.

Findings

NGSP outperforms existing region-based segmentation methods.

It maintains high performance with limited labeled data.

Effective on CAD shapes from online repositories.

Abstract

We propose the Neurally-Guided Shape Parser (NGSP), a method that learns how to assign fine-grained semantic labels to regions of a 3D shape. NGSP solves this problem via MAP inference, modeling the posterior probability of a label assignment conditioned on an input shape with a learned likelihood function. To make this search tractable, NGSP employs a neural guide network that learns to approximate the posterior. NGSP finds high-probability label assignments by first sampling proposals with the guide network and then evaluating each proposal under the full likelihood. We evaluate NGSP on the task of fine-grained semantic segmentation of manufactured 3D shapes from PartNet, where shapes have been decomposed into regions that correspond to part instance over-segmentations. We find that NGSP delivers significant performance improvements over comparison methods that (i) use regions to group per-point predictions, (ii) use regions as a self-supervisory signal or (iii) assign labels to regions under alternative formulations. Further, we show that NGSP maintains strong performance even with limited labeled data or noisy input shape regions. Finally, we demonstrate that NGSP can be directly applied to CAD shapes found in online repositories and validate its effectiveness with a perceptual study.