GeoSAM2: Unleashing the Power of SAM2 for 3D Part Segmentation

TL;DR

GeoSAM2 introduces a novel 3D part segmentation framework that uses multi-view 2D mask prediction guided by simple prompts, achieving state-of-the-art results without extensive training or labels.

Contribution

The paper presents GeoSAM2, a prompt-controllable 3D segmentation method that leverages SAM2 with multi-view 2D prompts, enabling fine-grained, interpretable, and efficient part segmentation.

Findings

Achieves state-of-the-art class-agnostic performance on PartObjaverse-Tiny and PartNetE.

Outperforms both optimization-based and coarse feedforward approaches.

Enables explicit, spatially grounded control without full 3D labels.

Abstract

We introduce GeoSAM2, a prompt-controllable framework for 3D part segmentation that casts the task as multi-view 2D mask prediction. Given a textureless object, we render normal and point maps from predefined viewpoints and accept simple 2D prompts - clicks or boxes - to guide part selection. These prompts are processed by a shared SAM2 backbone augmented with LoRA and residual geometry fusion, enabling view-specific reasoning while preserving pretrained priors. The predicted masks are back-projected to the object and aggregated across views. Our method enables fine-grained, part-specific control without requiring text prompts, per-shape optimization, or full 3D labels. In contrast to global clustering or scale-based methods, prompts are explicit, spatially grounded, and interpretable. We achieve state-of-the-art class-agnostic performance on PartObjaverse-Tiny and PartNetE, outperforming both slow optimization-based pipelines and fast but coarse feedforward approaches. Our results highlight a new paradigm: aligning the paradigm of 3D segmentation with SAM2, leveraging interactive 2D inputs to unlock controllability and precision in object-level part understanding.