Universal 3D Shape Matching via Coarse-to-Fine Language Guidance

TL;DR

UniMatch is a versatile 3D shape matching framework that uses language-guided semantic cues in a coarse-to-fine manner, enabling dense correspondences across diverse object categories without relying on isometry.

Contribution

It introduces a novel coarse-to-fine approach leveraging large language and vision models for universal, non-isometric 3D shape correspondence without predefined part proposals.

Findings

Outperforms existing methods in diverse scenarios

Effective across multiple object categories

No need for predefined part proposals

Abstract

Establishing dense correspondences between shapes is a crucial task in computer vision and graphics, while prior approaches depend on near-isometric assumptions and homogeneous subject types (i.e., only operate for human shapes). However, building semantic correspondences for cross-category objects remains challenging and has received relatively little attention. To achieve this, we propose UniMatch, a semantic-aware, coarse-to-fine framework for constructing dense semantic correspondences between strongly non-isometric shapes without restricting object categories. The key insight is to lift "coarse" semantic cues into "fine" correspondence, which is achieved through two stages. In the "coarse" stage, we perform class-agnostic 3D segmentation to obtain non-overlapping semantic parts and prompt multimodal large language models (MLLMs) to identify part names. Then, we employ pretrained vision language models (VLMs) to extract text embeddings, enabling the construction of matched semantic parts. In the "fine" stage, we leverage these coarse correspondences to guide the learning of dense correspondences through a dedicated rank-based contrastive scheme. Thanks to class-agnostic segmentation, language guiding, and rank-based contrastive learning, our method is versatile for universal object categories and requires no predefined part proposals, enabling universal matching for inter-class and non-isometric shapes. Extensive experiments demonstrate UniMatch consistently outperforms competing methods in various challenging scenarios.