Seg&Struct: The Interplay Between Part Segmentation and Structure Inference for 3D Shape Parsing

TL;DR

Seg&Struct introduces an integrated framework that jointly improves 3D shape part segmentation and structure inference by leveraging their mutual supervision and interplay, leading to significant performance gains.

Contribution

The paper presents a novel framework that fully exploits supervision to combine part segmentation and structure inference, enhancing both tasks through their interaction.

Findings

27.91% improvement in structure inference accuracy

0.5% improvement in segmentation accuracy

Demonstrates the synergy between segmentation and structural reasoning

Abstract

We propose Seg&Struct, a supervised learning framework leveraging the interplay between part segmentation and structure inference and demonstrating their synergy in an integrated framework. Both part segmentation and structure inference have been extensively studied in the recent deep learning literature, while the supervisions used for each task have not been fully exploited to assist the other task. Namely, structure inference has been typically conducted with an autoencoder that does not leverage the point-to-part associations. Also, segmentation has been mostly performed without structural priors that tell the plausibility of the output segments. We present how these two tasks can be best combined while fully utilizing supervision to improve performance. Our framework first decomposes a raw input shape into part segments using an off-the-shelf algorithm, whose outputs are then mapped to nodes in a part hierarchy, establishing point-to-part associations. Following this, ours predicts the structural information, e.g., part bounding boxes and part relationships. Lastly, the segmentation is rectified by examining the confusion of part boundaries using the structure-based part features. Our experimental results based on the StructureNet and PartNet demonstrate that the interplay between the two tasks results in remarkable improvements in both tasks: 27.91% in structure inference and 0.5% in segmentation.