PatchAlign3D: Local Feature Alignment for Dense 3D Shape understanding

TL;DR

PatchAlign3D introduces a 3D shape understanding model that directly produces language-aligned patch features from point clouds, enabling efficient zero-shot part segmentation without multi-view rendering.

Contribution

It presents a novel encoder-only 3D model trained with a new data pipeline, improving local part reasoning and zero-shot segmentation over prior multi-view, rendering-dependent methods.

Findings

Achieves state-of-the-art zero-shot 3D part segmentation performance.

Eliminates the need for multi-view rendering during inference.

Outperforms previous approaches on several 3D segmentation benchmarks.

Abstract

Current foundation models for 3D shapes excel at global tasks (retrieval, classification) but transfer poorly to local part-level reasoning. Recent approaches leverage vision and language foundation models to directly solve dense tasks through multi-view renderings and text queries. While promising, these pipelines require expensive inference over multiple renderings, depend heavily on large language-model (LLM) prompt engineering for captions, and fail to exploit the inherent 3D geometry of shapes. We address this gap by introducing an encoder-only 3D model that produces language-aligned patch-level features directly from point clouds. Our pre-training approach builds on existing data engines that generate part-annotated 3D shapes by pairing multi-view SAM regions with VLM captioning. Using this data, we train a point cloud transformer encoder in two stages: (1) distillation of dense 2D features from visual encoders such as DINOv2 into 3D patches, and (2) alignment of these patch embeddings with part-level text embeddings through a multi-positive contrastive objective. Our 3D encoder achieves zero-shot 3D part segmentation with fast single-pass inference without any test-time multi-view rendering, while significantly outperforming previous rendering-based and feed-forward approaches across several 3D part segmentation benchmarks. Project website: https://souhail-hadgi.github.io/patchalign3dsite/