PointCLIP: Point Cloud Understanding by CLIP

TL;DR

PointCLIP leverages CLIP's image-text alignment for 3D point cloud recognition by multi-view projection and adaptive fusion, enabling effective zero-shot and few-shot 3D understanding with minimal training.

Contribution

This work introduces PointCLIP, a novel method that adapts CLIP for 3D point cloud recognition through multi-view encoding and an inter-view adapter, achieving strong zero-shot and few-shot performance.

Findings

PointCLIP outperforms classical 3D-supervised networks in experiments.

Ensembling PointCLIP with traditional models boosts overall accuracy.

PointCLIP surpasses state-of-the-art models on ModelNet and ScanObjectNN datasets.

Abstract

Recently, zero-shot and few-shot learning via Contrastive Vision-Language Pre-training (CLIP) have shown inspirational performance on 2D visual recognition, which learns to match images with their corresponding texts in open-vocabulary settings. However, it remains under explored that whether CLIP, pre-trained by large-scale image-text pairs in 2D, can be generalized to 3D recognition. In this paper, we identify such a setting is feasible by proposing PointCLIP, which conducts alignment between CLIP-encoded point cloud and 3D category texts. Specifically, we encode a point cloud by projecting it into multi-view depth maps without rendering, and aggregate the view-wise zero-shot prediction to achieve knowledge transfer from 2D to 3D. On top of that, we design an inter-view adapter to better extract the global feature and adaptively fuse the few-shot knowledge learned from 3D into CLIP pre-trained in 2D. By just fine-tuning the lightweight adapter in the few-shot settings, the performance of PointCLIP could be largely improved. In addition, we observe the complementary property between PointCLIP and classical 3D-supervised networks. By simple ensembling, PointCLIP boosts baseline's performance and even surpasses state-of-the-art models. Therefore, PointCLIP is a promising alternative for effective 3D point cloud understanding via CLIP under low resource cost and data regime. We conduct thorough experiments on widely-adopted ModelNet10, ModelNet40 and the challenging ScanObjectNN to demonstrate the effectiveness of PointCLIP. The code is released at https://github.com/ZrrSkywalker/PointCLIP.