PartGlot: Learning Shape Part Segmentation from Language Reference Games

TL;DR

PartGlot is a neural framework that learns 3D shape part segmentation using only language descriptions and reference games, eliminating the need for explicit geometric annotations.

Contribution

It introduces a language-based learning approach for 3D shape segmentation that generalizes to unseen classes without direct geometric supervision.

Findings

The model accurately segments shape parts based on language references.

It generalizes to new shape classes not seen during training.

The approach reduces the need for large annotated datasets.

Abstract

We introduce PartGlot, a neural framework and associated architectures for learning semantic part segmentation of 3D shape geometry, based solely on part referential language. We exploit the fact that linguistic descriptions of a shape can provide priors on the shape's parts -- as natural language has evolved to reflect human perception of the compositional structure of objects, essential to their recognition and use. For training, we use the paired geometry / language data collected in the ShapeGlot work for their reference game, where a speaker creates an utterance to differentiate a target shape from two distractors and the listener has to find the target based on this utterance. Our network is designed to solve this target discrimination problem, carefully incorporating a Transformer-based attention module so that the output attention can precisely highlight the semantic part or parts described in the language. Furthermore, the network operates without any direct supervision on the 3D geometry itself. Surprisingly, we further demonstrate that the learned part information is generalizable to shape classes unseen during training. Our approach opens the possibility of learning 3D shape parts from language alone, without the need for large-scale part geometry annotations, thus facilitating annotation acquisition.