Parts2Words: Learning Joint Embedding of Point Clouds and Texts by Bidirectional Matching between Parts and Words

TL;DR

This paper introduces a novel method for shape-text matching that directly uses point clouds and bidirectional part-word matching, improving accuracy in multi-modal retrieval tasks.

Contribution

It proposes a joint embedding approach with part segmentation and optimal transport for shape-text matching, addressing limitations of view-based methods.

Findings

Significant accuracy improvement over state-of-the-art methods.

Effective joint embedding of point clouds and texts.

Robust shape-text matching demonstrated on Text2Shape dataset.

Abstract

Shape-Text matching is an important task of high-level shape understanding. Current methods mainly represent a 3D shape as multiple 2D rendered views, which obviously can not be understood well due to the structural ambiguity caused by self-occlusion in the limited number of views. To resolve this issue, we directly represent 3D shapes as point clouds, and propose to learn joint embedding of point clouds and texts by bidirectional matching between parts from shapes and words from texts. Specifically, we first segment the point clouds into parts, and then leverage optimal transport method to match parts and words in an optimized feature space, where each part is represented by aggregating features of all points within it and each word is abstracted by its contextual information. We optimize the feature space in order to enlarge the similarities between the paired training samples, while simultaneously maximizing the margin between the unpaired ones. Experiments demonstrate that our method achieves a significant improvement in accuracy over the SOTAs on multi-modal retrieval tasks under the Text2Shape dataset. Codes are available at https://github.com/JLUtangchuan/Parts2Words.