DenseMatcher: Learning 3D Semantic Correspondence for Category-Level Manipulation from a Single Demo

TL;DR

DenseMatcher is a novel method for 3D semantic correspondence that generalizes across object categories, enabling improved robotic manipulation and appearance transfer from a single demonstration.

Contribution

It introduces DenseMatcher, a new approach for 3D semantic correspondence, and provides the first 3D matching dataset with diverse colored object meshes.

Findings

Outperforms prior 3D matching methods by 43.5%.

Enables cross-category robotic manipulation from a single demo.

Facilitates zero-shot color transfer between objects.

Abstract

Dense 3D correspondence can enhance robotic manipulation by enabling the generalization of spatial, functional, and dynamic information from one object to an unseen counterpart. Compared to shape correspondence, semantic correspondence is more effective in generalizing across different object categories. To this end, we present DenseMatcher, a method capable of computing 3D correspondences between in-the-wild objects that share similar structures. DenseMatcher first computes vertex features by projecting multiview 2D features onto meshes and refining them with a 3D network, and subsequently finds dense correspondences with the obtained features using functional map. In addition, we craft the first 3D matching dataset that contains colored object meshes across diverse categories. In our experiments, we show that DenseMatcher significantly outperforms prior 3D matching baselines by 43.5%. We demonstrate the downstream effectiveness of DenseMatcher in (i) robotic manipulation, where it achieves cross-instance and cross-category generalization on long-horizon complex manipulation tasks from observing only one demo; (ii) zero-shot color mapping between digital assets, where appearance can be transferred between different objects with relatable geometry.