Point Clouds Are Specialized Images: A Knowledge Transfer Approach for 3D Understanding

TL;DR

This paper introduces PCExpert, a self-supervised learning method that treats point clouds as specialized images, enabling effective knowledge transfer from image models to improve 3D understanding tasks with fewer parameters.

Contribution

It proposes a novel approach to interpret point clouds as images, leveraging pre-trained image encoders and a new pretext task for enhanced 3D representation learning.

Findings

Achieves 90.02% accuracy on ScanObjectNN with linear fine-tuning.

Reduces the number of trainable parameters significantly.

Performance approaches full fine-tuning results, demonstrating robustness.

Abstract

Self-supervised representation learning (SSRL) has gained increasing attention in point cloud understanding, in addressing the challenges posed by 3D data scarcity and high annotation costs. This paper presents PCExpert, a novel SSRL approach that reinterprets point clouds as "specialized images". This conceptual shift allows PCExpert to leverage knowledge derived from large-scale image modality in a more direct and deeper manner, via extensively sharing the parameters with a pre-trained image encoder in a multi-way Transformer architecture. The parameter sharing strategy, combined with a novel pretext task for pre-training, i.e., transformation estimation, empowers PCExpert to outperform the state of the arts in a variety of tasks, with a remarkable reduction in the number of trainable parameters. Notably, PCExpert's performance under LINEAR fine-tuning (e.g., yielding a 90.02% overall accuracy on ScanObjectNN) has already approached the results obtained with FULL model fine-tuning (92.66%), demonstrating its effective and robust representation capability.