PointCLIP V2: Prompting CLIP and GPT for Powerful 3D Open-world Learning

TL;DR

PointCLIP V2 leverages prompt engineering with CLIP and GPT to enable powerful zero-shot and few-shot 3D open-world learning tasks such as classification, segmentation, and detection without domain-specific training.

Contribution

The paper introduces PointCLIP V2, a novel framework that unifies CLIP and GPT for enhanced zero-shot 3D understanding through innovative prompting and domain alignment techniques.

Findings

Achieves over 40% accuracy improvements in zero-shot 3D classification.

Effectively extends to few-shot classification, segmentation, and detection tasks.

Demonstrates strong generalization in 3D open-world learning scenarios.

Abstract

Large-scale pre-trained models have shown promising open-world performance for both vision and language tasks. However, their transferred capacity on 3D point clouds is still limited and only constrained to the classification task. In this paper, we first collaborate CLIP and GPT to be a unified 3D open-world learner, named as PointCLIP V2, which fully unleashes their potential for zero-shot 3D classification, segmentation, and detection. To better align 3D data with the pre-trained language knowledge, PointCLIP V2 contains two key designs. For the visual end, we prompt CLIP via a shape projection module to generate more realistic depth maps, narrowing the domain gap between projected point clouds with natural images. For the textual end, we prompt the GPT model to generate 3D-specific text as the input of CLIP's textual encoder. Without any training in 3D domains, our approach significantly surpasses PointCLIP by +42.90%, +40.44%, and +28.75% accuracy on three datasets for zero-shot 3D classification. On top of that, V2 can be extended to few-shot 3D classification, zero-shot 3D part segmentation, and 3D object detection in a simple manner, demonstrating our generalization ability for unified 3D open-world learning.