Best Segmentation Buddies for Image-Shape Correspondence

TL;DR

This paper introduces a novel method for establishing semantic correspondences between 2D image segments and 3D shape parts by distilling deep features and identifying Best Segmentation Buddies, demonstrating robustness across diverse pairs.

Contribution

The work presents a new approach to cross-modality segmentation correspondence by linking pixels to shape vertices through feature similarity and a novel buddy identification process.

Findings

Accurately establishes semantic correspondences across diverse image-shape pairs.

Effectively segments 3D shapes directly using distilled 2D features.

Demonstrates robustness and generality in challenging real-world scenarios.

Abstract

Finding correspondences is a fundamental and extensively researched problem in computer vision and graphics. In this work, we examine the underexplored task of estimating segmentation-to-segmentation correspondence between images in the wild and untextured 3D shapes. This task is highly challenging due to substantial differences in appearance, geometry, and viewpoint. Our approach bridges the cross-modality gap by linking pixels in the image segment to vertices in the corresponding semantic part of the 3D shape. To achieve this, we first distill deep visual features from a 2D vision model onto the 3D shape surface, allowing for the computation of feature similarity between image pixels and shape vertices. Then, we identify Best Segmentation Buddies, vertices whose most similar image pixel lies within the image segmentation region, enabling the reliable discovery of vertices in semantically corresponding shape parts. Finally, we leverage distilled 3D features from the 2D image segmentation model to segment the shape directly in 3D, bootstrapping the correspondence process. We demonstrate the generality and robustness of our approach across a wide range of image-shape pairs, showcasing accurate and semantically meaningful correspondences. Our project page is at https://threedle.github.io/bsb/.