Adapt PointFormer: 3D Point Cloud Analysis via Adapting 2D Visual Transformers

TL;DR

This paper introduces Adaptive PointFormer, a novel method that leverages pre-trained 2D visual transformers for 3D point cloud analysis by converting raw point clouds into embeddings and fine-tuning with minimal parameters.

Contribution

It proposes a new approach to adapt pre-trained 2D models for 3D point cloud tasks without converting data into images, reducing computational costs and maintaining effectiveness.

Findings

Effective on multiple benchmarks

Outperforms existing 3D point cloud methods

Uses minimal additional parameters

Abstract

Pre-trained large-scale models have exhibited remarkable efficacy in computer vision, particularly for 2D image analysis. However, when it comes to 3D point clouds, the constrained accessibility of data, in contrast to the vast repositories of images, poses a challenge for the development of 3D pre-trained models. This paper therefore attempts to directly leverage pre-trained models with 2D prior knowledge to accomplish the tasks for 3D point cloud analysis. Accordingly, we propose the Adaptive PointFormer (APF), which fine-tunes pre-trained 2D models with only a modest number of parameters to directly process point clouds, obviating the need for mapping to images. Specifically, we convert raw point clouds into point embeddings for aligning dimensions with image tokens. Given the inherent disorder in point clouds, in contrast to the structured nature of images, we then sequence the point embeddings to optimize the utilization of 2D attention priors. To calibrate attention across 3D and 2D domains and reduce computational overhead, a trainable PointFormer with a limited number of parameters is subsequently concatenated to a frozen pre-trained image model. Extensive experiments on various benchmarks demonstrate the effectiveness of the proposed APF. The source code and more details are available at https://vcc.tech/research/2024/PointFormer.