3x2: 3D Object Part Segmentation by 2D Semantic Correspondences

TL;DR

This paper introduces 3-By-2, a novel method for 3D object part segmentation that leverages 2D semantic correspondences and pretrained models to overcome limited 3D annotations, achieving state-of-the-art results.

Contribution

The paper presents a new approach that uses 2D datasets and pretrained features to perform 3D segmentation, enabling flexible part taxonomies and label transfer.

Findings

Achieves SOTA performance on multiple benchmarks.

Effectively leverages 2D annotations for 3D segmentation.

Demonstrates cross-category part label transfer.

Abstract

3D object part segmentation is essential in computer vision applications. While substantial progress has been made in 2D object part segmentation, the 3D counterpart has received less attention, in part due to the scarcity of annotated 3D datasets, which are expensive to collect. In this work, we propose to leverage a few annotated 3D shapes or richly annotated 2D datasets to perform 3D object part segmentation. We present our novel approach, termed 3-By-2 that achieves SOTA performance on different benchmarks with various granularity levels. By using features from pretrained foundation models and exploiting semantic and geometric correspondences, we are able to overcome the challenges of limited 3D annotations. Our approach leverages available 2D labels, enabling effective 3D object part segmentation. Our method 3-By-2 can accommodate various part taxonomies and granularities, demonstrating interesting part label transfer ability across different object categories. Project website: \url{https://ngailapdi.github.io/projects/3by2/}.