Adaptive Point-Prompt Tuning: Fine-Tuning Heterogeneous Foundation Models for 3D Point Cloud Analysis

TL;DR

This paper introduces Adaptive Point-Prompt Tuning (APPT), a parameter-efficient method for adapting heterogeneous foundation models to 3D point cloud analysis by directly leveraging point features without complex mappings.

Contribution

It proposes a novel fine-tuning framework that uses point prompts and permutation-invariant features to adapt pre-trained models for 3D point cloud tasks efficiently.

Findings

Enables direct point cloud processing with minimal parameters.

Preserves spatial geometry through local aggregation and permutation-invariant features.

Reduces computational overhead with a shared prompt generator.

Abstract

Parameter-efficient fine-tuning strategies for foundation models in 1D textual and 2D visual analysis have demonstrated remarkable efficacy. However, due to the scarcity of point cloud data, pre-training large 3D models remains a challenging task. While many efforts have been made to apply pre-trained visual models to 3D domains through "high-to-low" mapping, these approaches often lead to the loss of spatial geometries and lack a generalizable framework for adapting any modality to 3D. This paper, therefore, attempts to directly leverage point features to calibrate the heterogeneous foundation model of any modality for 3D point cloud analysis. Specifically, we propose the Adaptive Point-Prompt Tuning (APPT) method, which fine-tunes pre-trained models with a modest number of parameters, enabling direct point cloud processing without heterogeneous mappings. We convert raw point clouds into point embeddings by aggregating local geometry to capture spatial features followed by linear layers to ensure seamless utilization of frozen pre-trained models. Given the inherent disorder of point clouds, in contrast to the structured nature of images and language, we employ a permutation-invariant feature to capture the relative positions of point embeddings, thereby obtaining point tokens enriched with location information to optimize self-attention mechanisms. To calibrate self-attention across source domains of any modality to 3D and reduce computational overhead, we introduce a prompt generator that shares weights with the point embedding module, dynamically producing point-prompts without adding additional parameters. These prompts are then concatenated into a frozen foundation model, providing rich global structural information and compensating for the lack of structural context in the heterogeneous data.

Figures

Tables

Table 1. TABLE I: Comparisons on accuracy for object classification on ScanObjectNN and ModelNet40. The best and second-best results are highlighted in underlined bold and bold , respectively. The superscript * denotes results obtained using ViT-B for P2P to ensure a fair comparison. “Aud.” is an abbreviation for “Audio”.

Methods	Published	Pretrained	ScanObjectNN			ModelNet40
	Year	Modality	OBJ-BG	OBJ-ONLY	PB-T50-RS
MLP/CNN-based Model
PointNet [28]	2017	N/A	73.8	79.2	68.0	89.2
DGCNN [34]	2019	N/A	82.8	86.2	78.1	92.9
PointMLP [62]	2022	N/A	-	-	85.2	94.1
Point-PN [63]	2023	N/A	91.0	90.2	87.1	93.8
PointNet-OcCo [13]	2021	3D	-	-	80.0	90.1
DGCNN-OcCo [13]	2021	3D	-	-	83.9	93.0
MHSA-based Model
Transformer [36]	2017	N/A	79.9	80.6	77.2	91.4
PCT [38]	2021	N/A	-	-	-	93.2
Transformer-OcCo [13]	2021	3D	84.9	85.5	78.8	92.1
Point-BERT [12]	2022	3D	87.4	88.1	83.1	93.2
Point-MAE [49]	2022	3D	90.0	88.3	85.2	93.8
Joint-MAE [74]	2023	3D	90.9	88.9	86.1	94.0
Point-BERT w. Point-PEFT [48]	2024	3D	-	-	85.0	93.4
Point-BERT w. DAPT [75]	2024	3D	91.1	89.7	85.4	93.6
P2P∗ [16]	2022	2D	-	-	84.1	92.4
APF [27]	2024	2D	89.9	89.0	87.8	94.2
Any2Point [15]	2024	2D	-	-	87.7	93.2
APPT	Ours	2D	92.4	90.5	92.6	94.2
ACT [56]	2023	3D+2D	87.1	89.0	81.5	93.7
ReCon [76]	2023	3D+2D+ 1D (Text)	90.6	90.7	83.8	93.4
Any2Point [15]	2024	1D (Aud.)	-	-	87.0	92.7
Any2Point [15]	2024	1D (Text)	-	-	91.9	94.3
APPT	Ours	1D (Aud.)	92.3	90.7	88.9	94.6
APPT	Ours	1D (Text)	91.9	90.2	91.4	95.1

Table 2. TABLE II: Few-shot classification results on ModelNet40.

Methods	Pre-trained Modality	5-way		10-way
		10-shot	20-shot	10-shot	20-shot
MLP/CNN-based Model
PointNet [28]	N/A	52.0 $\pm$ 3.8	57.8 $\pm$ 4.9	46.6 $\pm$ 4.3	35.2 $\pm$ 4.8
PointNet-OcCo [13]	3D	89.7 $\pm$ 1.9	92.4 $\pm$ 1.6	83.9 $\pm$ 1.8	89.7 $\pm$ 1.5
PointNet w. CrossPoint [77]	2D	90.9 $\pm$ 4.8	93.5 $\pm$ 4.4	84.6 $\pm$ 4.7	90.2 $\pm$ 2.2
DGCNN [34]	N/A	31.6 $\pm$ 2.8	40.8 $\pm$ 4.6	19.9 $\pm$ 2.1	16.9 $\pm$ 1.5
DGCNN-OcCo [13]	3D	90.6 $\pm$ 2.8	92.5 $\pm$ 1.9	82.9 $\pm$ 1.3	86.5 $\pm$ 2.2
DGCNN w. CrossPoint [77]	2D	92.5 $\pm$ 3.0	94.9 $\pm$ 2.1	83.6 $\pm$ 5.3	87.9 $\pm$ 4.2
MHSA-based Model
Transformer [36]	N/A	87.8 $\pm$ 5.2	93.3 $\pm$ 4.3	84.6 $\pm$ 5.5	89.4 $\pm$ 6.3
Transformer-OcCo [13]	3D	94.0 $\pm$ 3.6	95.9 $\pm$ 2.3	89.4 $\pm$ 5.1	92.4 $\pm$ 4.6
Point-BERT [12]	3D	94.6 $\pm$ 3.1	96.3 $\pm$ 2.7	91.0 $\pm$ 5.4	92.7 $\pm$ 5.1
Point-MAE [49]	3D	96.3 $\pm$ 2.5	97.8 $\pm$ 1.8	92.6 $\pm$ 4.1	95.0 $\pm$ 3.0
Joint-MAE [74]	3D	96.7 $\pm$ 2.2	97.9 $\pm$ 1.8	92.6 $\pm$ 3.7	95.1 $\pm$ 2.6
Point-BERT w. DAPT [75]	3D	95.8 $\pm$ 2.1	97.3 $\pm$ 1.3	92.2 $\pm$ 4.3	94.2 $\pm$ 3.4
APF [46]	2D	96.9 $\pm$ 1.8	98.1 $\pm$ 1.8	92.6 $\pm$ 2.4	95.7  $\pm$ 1.6
APPT (ours)	2D	97.0 $\pm$ 1.0	99.1 $\pm$ 0.9	92.7 $\pm$ 0.8	95.3  $\pm$ 2.3
APPT (ours)	1D (Text)	96.5 $\pm$ 2.0	99.0 $\pm$ 1.0	91.5 $\pm$ 2.5	95.1  $\pm$ 2.1
APPT (ours)	1D (Aud.)	96.5 $\pm$ 1.5	99.1 $\pm$ 0.9	91.4 $\pm$ 1.6	94.9  $\pm$ 1.9

Table 3. TABLE III: Part segmentation results on ShapeNetPart. mIoU C \text{mIoU}_{C} (%) is the mean of class IoU. mIoU I \text{mIoU}_{I} (%) is the mean of instance IoU. “Trans.” abbreviates for Transformer.

Methods	${\text{mIoU}}_C$	${\text{mIoU}}_I$	aero-plane	bag	cap	car	chair	ear-phone	guitar	knife	lamp	laptop	motor-bike	mug	pistol	rocket	skate-board	table
MLP/CNN-based Model
PointNet [28]	80.4	83.7	83.4	78.7	82.5	74.9	89.6	73.0	91.5	85.9	80.8	95.3	65.2	93.0	81.2	57.9	72.8	80.6
PointNet++ [25]	81.9	85.1	82.4	79.0	87.7	77.3	90.8	71.8	91.0	85.9	83.7	95.3	71.6	94.1	81.3	58.7	76.4	82.6
DGCNN [34]	82.3	85.2	84.0	83.4	86.7	77.8	90.6	74.7	91.2	87.5	82.8	95.7	66.3	94.9	81.1	63.5	74.5	82.6
KPConv [35]	85.1	86.4	84.6	86.3	87.2	81.1	91.1	77.8	92.6	88.4	82.7	96.2	78.1	95.8	85.4	69.0	82.0	83.6
PAConv [81]	84.6	86.1	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
PointMLP [62]	84.6	86.1	83.5	83.4	87.5	80.54	90.3	78.2	92.2	88.1	82.6	96.2	77.5	95.8	85.4	64.6	83.3	84.3
MHSA-based Model
Trans. [36]	83.4	85.1	82.9	85.4	87.7	78.8	90.5	80.8	91.1	87.7	85.3	95.6	73.9	94.9	83.5	61.2	74.9	80.6
Point Trans. [37]	83.7	86.6	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
PCT [38]	-	86.4	85.0	82.4	89.0	81.2	91.9	71.5	91.3	88.1	86.3	95.8	64.6	95.8	83.6	62.2	77.6	83.7
Trans.-OcCo [13]	83.4	85.1	83.3	85.2	88.3	79.9	90.7	74.1	91.9	87.6	84.7	95.4	75.5	94.4	84.1	63.1	75.7	80.8
Point-BERT [12]	84.1	85.6	84.3	84.8	88.0	79.8	91.0	81.7	91.6	87.9	85.2	95.6	75.6	94.7	84.3	63.4	76.3	81.5
Point-MAE [49]	-	86.1	84.3	85.0	88.3	80.5	91.3	78.5	92.1	87.4	96.1	96.1	75.2	94.6	84.7	63.5	77.1	82.4
P2P∗ [16]	82.5	85.7	83.2	84.1	85.9	78.0	91.0	80.2	91.7	87.2	85.4	95.4	69.6	93.5	79.4	57.0	73.0	83.6
Joint-MAE [74]	85.4	86.3	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
Point-BERT [12] w. DAPT [75]	83.8	85.5	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
APF [27]	83.4	86.1	83.6	84.8	85.4	79.8	91.3	77.0	91.4	88.4	84.4	95.5	76.3	95.3	82.5	59.5	76.1	83.5
APPT (ours)	84.0	85.9	83.5	85.0	86.7	79.8	91.9	79.6	91.9	87.9	83.7	96.1	76.2	95.8	82.2	65.1	76.4	82.8

Table 4. TABLE IV: Impact of each component. The abbreviations are defined as follows: PE: pint embedding, 2D Mod.: 2D modality, PPT: point prompt tuning, SONN: ScanObjectNN, and MN40: ModelNet40.

PE	2D Mod.	PosIn	PPT	SONN	MN40
				Acc. (%)	Acc. (%)
✓	✗	✗	✗	87.1 (base)	93.8 (base)
✓	✓	✗	✗	90.1 ($\uparrow$ 3.0)	93.9 ($\uparrow$ 0.1)
✓	✓	✓	✗	91.2 ($\uparrow$ 4.1)	94.1 ($\uparrow$ 0.3)
✓	✓	✗	✓	91.4 ($\uparrow$ 4.3)	94.1 ($\uparrow$ 0.3)
✓	✓	✓	✓	92.6 ($\uparrow$ 5.5)	94.2 ($\uparrow$ 0.4)

Table 5. TABLE V: Performance comparison of different pre-trained modality (Pre. Mod.) on ScanObjectNN (PB-T50-RS).

Method	Pre. Mod.	Model	Acc. (%)
APF [27]	2D	ViT-B [8]	87.8
	2D	DINOv2 [10]	87.7
	2D	DeiT [84]	87.3
Any2Point [15]	1D (Aud.)	ImageBind [83]	87.0
	1D (Text)	CLIP [9]	91.9
	1D (Text)	RoBERTa [85]	89.7
	2D	ViT-B [8]	92.6
	2D	DINOv2 [10]	92.6
APPT (Ours)	2D	DeiT [84]	88.9
	1D (Aud.)	ImageBind [83]	88.9
	1D (Text)	CLIP [9]	91.4
	1D (Text)	RoBERTa [85]	87.3

Table 6. TABLE VI: Comparison results of different pre-trained models on ScanObjectNN PB-T50-RS (SONN) and ModelNet (MN40) datasets.

Method	Model	SONN	MN40
		Acc. (%)	Acc. (%)
Point-PN	N/A	87.1	93.8
Transformer	N/A	77.2	91.4
APPT w. 2D	ViT-B	92.6 ($\uparrow$ 5.5)	94.2 ($\uparrow$ 0.4)
APPT w. 1D	CLIP	91.4 ($\uparrow$ 4.3)	95.1 ($\uparrow$ 1.3)

Table 7. TABLE VII: Performance comparison w.r.t. trainable parameters number (# Tr. param.) on ScanObjectNN (PB-T50-RS).

Method	Pre. Mod.	# Tr. Param.	Acc. (%)
PointNet++	N/A	1.4M	77.9
PointMLP	N/A	12.6M	85.2
DGCNN-OcCo	3D	1.8M	83.9
Point-BERT	3D	21.1M	83.1
Point-MAE	3D	21.1M	85.2
P2P w. ViT-B	2D	0.25M	84.1
P2P w. HorNet-L-22k-mlp	2D	1.2M	89.3
Any2Point	2D	0.8M	87.7
APF w. PointNet	2D	2.4M	83.1
APF w. PointMLP	2D	5.8M	87.8
APPT (ours)	2D	3.4M	92.6

Equations

$$e_i^{H,(0)}$$

$$\left[e_{\text{cls}}^{H,(l)},{\mathcal{E}}^{H,(l)}\right]$$

$$\{\begin{array}{l}{\tilde{e}}_i^{H,(l)}={\text{MHSA}}^l\left(e_i^{H,(l-1)}\right)+e_i^{H,(l-1)}\\ e_i^{H,(l)}={\text{MLP}}^l\left(\text{LN}\left({\tilde{e}}_i^{H,(l)}\right)\right)+{\tilde{e}}_i^{H,(l)}\end{array}.$$

$${\hat{e}}_i^P=\text{Point\_Embed}\left({\mathcal{X}}_i^P\right),$$

$$e_g=\text{avgP}\left({\hat{\mathcal{E}}}^P\right).$$

$$e_i^{P,(0)}=a\cdot ({\hat{e}}_i^P-e_g)+b\cdot {\hat{e}}_i^P,$$

$$e_i^{P,(0)}=\text{Conv1D}\left(\text{Concat}\{{\hat{e}}_i^P-e_g,{\hat{e}}_i^P\}\right),$$

$$p_0=\text{maxP}\left({\hat{\mathcal{E}}}^P\right)+\text{avgP}\left({\hat{\mathcal{E}}}^P\right),$$

$${\mathcal{E}}^{P,(0)}=\text{PosIn}\left(\text{Point\_Embed}\left(\left[{\mathcal{X}}_i^p\right]\right)\right),$$

$$\displaystyle\left[e_{\text{cls}}^{P,(1)},\mathcal{Z}^{(1)},\mathcal{E}^{P,(1)}\right]=\phi^{(1)}\left(\left[{\color[rgb]{0.28125,0.453125,0.796875}e_{\text{cls}}^{P,(0)}},{\color[rgb]{0.78515625,0.11328125,0.19140625}p_{0}},{\color[rgb]{0.78515625,0.11328125,0.19140625}\mathcal{E}^{P,(0)}}\right]\right),$$

$$\displaystyle\left[e_{\text{cls}}^{P,(l)},\mathcal{Z}^{(l)},\mathcal{E}^{P,(l)}\right]=\phi^{(l)}\left(\left[{\color[rgb]{0.28125,0.453125,0.796875}e_{\text{cls}}^{P,(l-1)}},\left[{\color[rgb]{0.78515625,0.11328125,0.19140625}p_{0}},{\color[rgb]{0.28125,0.453125,0.796875}\mathcal{Z}^{(l-1)}}\right],{\color[rgb]{0.28125,0.453125,0.796875}\mathcal{E}^{P,(l-1)}}\right]\right),$$

$$e_{\text{cls}}=\text{Pool}\left(\left[e_{\text{cls}}^{P,(L)},{\mathcal{Z}}^{(L)},{\mathcal{E}}^{P,(L)}\right]\right),$$

$$p_i=\frac{e^{w_i\cdot e_{\text{cls}}}}{{\sum }_{j=1}^C{e}^{w_j\cdot e_{\text{cls}}}},$$

$$g(\mathbf{S})=g(\sigma (\mathbf{S})),$$

$$\text{mean}\left(\sigma (S)\right)=\frac{1}{n}\sum _{i=1}^n{s}_{\sigma (i)}.$$

$$\frac{1}{n}\sum _{i=1}^n{s}_{\sigma (i)}=\frac{1}{n}\sum _{i=1}^n{s}_i$$

$$\text{mean}(\sigma (S))=\text{mean}(S).$$

Full text

1 Introduction

Parameter-efficient fine-tuning (PEFT) [1, 2, 3, 4] has emerged as a widely adopted strategy for leveraging the rich semantic features and representation capabilities of large foundation models across diverse downstream tasks, while simultaneously reducing computational and storage costs [5]. This progress has been particularly notable in the fields of natural language processing (NLP) [6, 7] and computer vision (CV) [8, 9, 10], where the growing availability of training data has led to the continuous emergence of pre-trained foundation models. However, 3D visual understanding [11], as an important research topic, faces significantly greater challenges in data acquisition compared to NLP and CV. This results in a lack of large-scale foundation models for 3D tasks. Although several 3D pre-trained models, such as Point-BERT [12], OcCo [13], and PointGPT [14], have shown promising results, their scale remains incomparable to models trained on image or text data. For instance, the 3D foundation model, PointGPT-L [14] is pre-trained on a multi-source dataset containing approximately 3 million point clouds, whereas the visual-linguistic model CLIP [9] is trained on 400 million image-text pairs. Acquiring and annotating real high-quality 3D data requires significant resources and human labor, and synthetic 3D data often lacks distribution diversity and real-world applicability [15]. These limitations raise the question of whether prior knowledge from 2D or 1D data can be effectively leveraged for the analysis of 3D point clouds.

Previous work has demonstrated the feasibility of transferring prior knowledge from heterogeneous data to 3D point cloud analysis, typically following two main routes.

1. Modality projection [16, 17, 18, 19, 15] involves projecting 3D point clouds into lower-dimensional modalities, such as 1D linguistic or 2D visual representations, to leverage the pre-trained foundation models. However, directly projecting 3D point clouds onto 1D/2D data inevitably results in the loss of high-dimensional information. Recently, Tang et al. [15] have proposed Any2Point, which virtually projects 3D coordinates to 2D (or 1D) space to utilize the position embedding of pre-trained large models. This approach mitigates the issue of dimensional information loss by assigning positional embeddings compatible with the pre-trained model to 3D tokens. Nonetheless, it still relies on low-dimensional projections to exploit prior knowledge and does not fully adapt the pre-trained self-attention mechanism to the 3D domain.
2. Knowledge distillation [20, 21, 22, 23, 24] facilitates the training of specialized 3D models by transferring knowledge from pre-trained models trained on heterogeneous data. However, these methods not only require training 3D models from scratch but also heavily rely on large-scale paired 2D and/or 1D-3D data. Their data dependencies require extensive engineering efforts, ultimately limiting their efficiency and generalization capacity.

To address these challenges, we propose a novel approach, Adaptive Point-Prompt Tuning (APPT), which directly leverages point features to adapt heterogeneous foundation models to the 3D modality, thereby optimizing the utilization of high-dimensional point cloud information while reducing computational costs. In contrast to modality projection methods, the proposed APPT, as shown in Fig. 1, directly processes point clouds and effectively preserves 3D information. Specifically, APPT encodes point embeddings using farthest point sampling, k-nearest neighbors, pooling operations [25], and local geometry aggregation to effectively handle unordered data and capture spatial features. A linear operation is incorporated into a point embedding module to calibrate dimensionality, ensuring seamless alignment with pre-trained large models. To enhance robustness against point permutations and capture geometric and semantic relationships between point embeddings, we exploit permutation-invariance [26] for relative position injection into the token generation process. The prompt tuning strategy [4] is employed to adaptively fine-tune the self-attention mechanism in pre-trained models. Notably, the prompt is a global representation generated by a point generator that shares weights with the point embedding module, followed by a pooling operation. As a result, the point embedding module is the only trainable component, facilitating the adaptation of pre-trained models from source modalities without the need to train an entire 3D network, thereby significantly enhancing computational efficiency. By integrating point cloud information with the heterogeneous semantic priors from pre-trained models, APPT effectively addresses a variety of downstream 3D point cloud analysis tasks. Extensive experiments on benchmark datasets demonstrate that the proposed APPT consistently surpasses the existing methods across various downstream tasks. In summary, our main contributions are as follows.

• •

We investigate the potential of pre-trained models on heterogeneous data for 3D point cloud analysis without dimension reduction and propose the APPT method to effectively leverage such models. Our method demonstrates that rich 2D or 1D priors can offer valuable knowledge for the 3D domain, and with minimal fine-tuning, it can outperform models trained exclusively on 3D data.

• •

We propose a position injector (PosIn) that encodes position information with negligible training parameters. The concept of permutation-invariant features is introduced to identify an embedding centroid, ensuring invariance across tokens and allowing the model to remain unaffected by the order of points and tokens. PosIn directs the model focus on underlying relationships and dependencies, rather than the order of points, thereby enhancing the applicability of pre-trained models.

• •

We propose a novel point-prompt generator that shares weights with the point embedding module and includes a permutation-invariant operation for obtaining order-independent global representations. This generator enables direct fine-tuning of heterogeneous pre-trained models for point cloud analysis, eliminating the need for lossy mappings or time-consuming training.

• •

The proposed APPT outperforms the existing methods, as demonstrated through extensive experiments on a variety of 3D downstream tasks. These experiments utilize a range of pre-trained large models, including both linguistic and visual models, consistently achieving superior performance while fine-tuning only 3.8% of the parameters.

A preliminary version of this work has been published in [27]. This paper has four major improvements. First, we enhance the point embedding module by incorporating local geometry aggregation and linear operations, instead of using the entire PointNet or PointMLP. This approach better handles unordered data, captures richer contextual information, and further reduces computational overhead. Second, we improve the fine-tuning strategy with the prompt generator, making it a plug-and-play module compatible with various pre-trained models. Third, we replace the sequencing operation in [27] with a position injector that has permutation-invariance property across point embeddings to enhance the feature representation of point clouds and mitigate the impact of irrelevant location information. The ablation study demonstrates that the simple yet effective modules in APPT lead to significant improvements. Finally, we extend the foundation model from a 2D-only model to various pre-trained foundation models, including visual, textual, and audio models, to demonstrate the effectiveness and generalization of the proposed method across diverse pre-trained knowledge sources. The proposed APPT consistently outperforms existing state-of-the-art methods.

2 Related work

2.1 MLP/CNN-based 3D Specialized Model

Since the introduction of PointNet [28], deep learning-based approaches for point cloud processing have experienced rapid development in recent years. These methods can be categorized into three groups based on the representations of point clouds: voxel-based [29, 30], projection-based [31, 32], and point-based [11, 33]. Voxel-based methods entail the voxelization of input points into regular voxels, utilizing CNNs for subsequent processing. However, these methods tend to incur substantial memory consumption and slower runtime, particularly when a finer-grained representation is required [11]. Projection-based methods involve converting point clouds into dense 2D grids, which are then treated as a regular image. This transformation enables the application of classical image-processing techniques to tackle challenges in point cloud analysis. However, these methods heavily rely on projection and back-projection processes, presenting challenges, particularly in urban scenes with diverse scales in different directions. In contrast, point-based methods, directly applied to 3D point clouds, are the most widely adopted. Such methods commonly employ shared multi-layer perceptrons or incorporate sophisticated convolution operators [28, 25, 34, 35]. In recent years, hybrid methods such as PVCNN [29] and PV-RCNN [30], which combine the strengths of diverse techniques, have achieved notable advancements.

2.2 Self-Attention-based Specialized 3D Model

Self-attention operations [36] have been adopted for point cloud processing in several studies [37, 38, 39]. The point Transformer [37] and point cloud Transformer (PCT) [38] have introduced self-attention networks [36] to improve the capture of local context within the point clouds. Afterward, a plethora of methods based on the self-attention architecture have been proposed, which can be categorized into point-based [39, 40, 41, 42, 43], heterogeneous auxiliary information-based [44, 45], and homogeneous auxiliary information-based [46, 47, 48] methods. Point-based methods structure point clouds by sorting them according to specific patterns, transforming unstructured, irregular point clouds into manageable sequences while preserving spatial proximity. This approach emulates token sequences in NLP, allowing the use of the self-attention mechanism. Heterogeneous auxiliary information-based methods integrate supplementary data from diverse sources (e.g., images, semantic labels) to enhance the understanding and performance of 3D point cloud tasks through multi-modal fusion and cross-modal learning techniques. For example, tokenFusion [44] initially fuses tokens from point clouds and images, subsequently forwarding the fused tokens to a shared Transformer network, allowing the learning of correlations among multimodal features. However, these methods suffer from high memory consumption and computational complexity [40], as they require training the entire network from scratch. Homogeneous auxiliary information-based methods introduce 3D pre-trained models. By fine-tuning existing pre-trained models, their performance on 3D-related tasks can be significantly improved, while computational costs can be effectively reduced. For example, Point-Bert [12], Point-MAE [49], and PointM2AE [50] integrate masking techniques with pre-trained 3D models, enhancing the ’s generalization of models to unseen data while requiring less task-specific training. However, compared to image data, point cloud data is more difficult to acquire, and the capability of 3D pre-trained models is relatively weaker.

2.3 Point Cloud Analysis with 2D Foundation Model

Leveraging knowledge from 2D to 3D seeks to strengthen the 3D understanding and improve the accuracy of 3D downstream tasks by utilizing the rich contextual information and prior knowledge embedded in pre-trained 2D models. Most current research [16, 18, 17, 51, 52, 53] relies on 3D-to-2D projection. In this approach, the tokens derived from the 3D point cloud data are projected onto 2D planes, after which an existing 2D pre-trained model is employed to efficiently process the projected tokens. While this method has proven effective, projecting 3D data to 2D introduces several challenges, such as the loss of 3D spatial information, limited handling of complex geometries, and dependency on projection angles [54, 15], to name a few. To address these issues, several studies focus on minimizing the information loss from high-dimensional to low-dimensional representations. For example, Any2Point [15] proposes a virtual projection technique to map point clouds onto 1D or 2D planes. Nevertheless, these methods still cannot directly process 3D data. Cross-modality knowledge distillation methods [55, 23, 56, 24] typically transfer the knowledge learned by a 2D model to a smaller 3D network, enabling data-efficient training while being 3D-specific. The 3D model benefits from the rich prior knowledge acquired by the 2D/1D model. For example, ACT [56] employs pre-trained visual or language models to assist in 3D representation learning, serving as a cross-modal teacher, which enables the student model for point clouds to be trained with enhanced representational capacity. ULIP [23] and ULIP-2 [24] leverage the vision-language models pre-trained on large-scale image-text pairs, aligning the feature space of a point cloud encoder with the pre-aligned vision/language feature space. However, the dependence on paired 1D/2D-3D data limits the flexibility of these methods.

3 Adaptive Point-Prompt Tuning

We propose adaptive point-prompt tuning (APPT), a method to adapt large-scale Transformer-based models pre-trained on heterogeneous data, for downstream tasks in the 3D point cloud modality. The raw point cloud grouping input into APPT and the Transformer encoder structure employed in our approach are first overviewed in Sec. 3.1. APPT encodes the input point groups into point tokens using the point embedding and position injector modules in Sec. 3.2, ensuring that the dimensionality and position information of the point tokens match those of the input tokens in the pre-trained Transformer models. Subsequently, the prior self-attention mechanism of the foundation model is adapted by injecting a point-prompt, generated by a learnable prompt generator as described in Sec. 3.3, while keeping the backbone frozen during the downstream training phase. The token embedding module and prompt generator share knowledge to ensure consistent feature representation and reduce the number of trainable parameters. To enhance the structural knowledge of the point token encoding and allow for more effective information flow, APPT propagates the features encoded by each block to the next, as detailed in Sec. 3.4. This contrasts with existing fine-tuning strategies, such as VPT-shallow and VPT-deep [57], where trainable prompts are inserted only into the first or each Transformer block without being passed to subsequent blocks. The overall pipeline of APPT is illustrated in Fig. 2. Sec. 3.5 explains the rationale behind the proposed APPT, demonstrating its effectiveness in capturing spatial structure and global features from 3D data to provide valuable information for fine-tuning pre-trained models.

3.1 Preliminaries

Raw Point Grouping. Given the input point clouds $\mathcal{P}\in\mathbb{R}^{N\times{(d^{\prime}+C)}}$, where $N$ represents the number of unordered points, denoted as $\mathcal{P}=\left[x^{P}_{1},x^{P}_{2},\cdots,x^{P}_{N}\right]$ and $x^{P}_{i}\in\mathbb{R}^{d^{\prime}+C}$ with $d^{\prime}$-dim coordinates and $C$-dim point feature, we first employ iterative farthest point sampling (FPS) to sample a subset of points $\mathcal{P}_{s}=\left[x^{P}_{1},x^{P}_{2},\cdots,x^{P}_{N_{s}}\right]\in\mathbb{R}^{N_{s}\times(d^{\prime}+C)}$. Subsequently, the $k$-nearest neighbors $\mathcal{P}_{g}=\left[\left\{x^{P}_{1,j}\right\}_{j=1}^{k},\left\{x^{P}_{2,j}\right\}_{j=1}^{k},\cdots,\left\{x^{P}_{N_{s},j}\right\}_{j=1}^{k}\right]\in\mathbb{R}^{N_{s}\times k\times(d^{\prime}+C)}$ for each point are identified, wherein each group $\left\{x^{P}_{i,j}\right\}_{j=1}^{k}$ within $\mathcal{P}_{g}$ corresponds to a local region around the centroid point $x^{P}_{i}$, and $k$ represents the number of points adjacent to the $N_{s}$ centroid points. Following this, embedding $\mathcal{P}_{g}$ becomes necessary to leverage the heterogeneous priors embedded in pre-trained models.

Transformer Encoder.

The transformer [36] encoder comprises an embedding layer and multiple transformer blocks. For a non-point cloud input $x^{H}$, which can be a sentence [6], an image [58] or speech [59], the model first partitions $x^{H}$ into $m$ patches, forming a set $\{x^{H}_{i}\}_{i=1}^{m}$. These patches are then embedded into sequences of $d^{H}$-dimensional vectors, denoted as $\mathcal{E}^{H}_{0}=\texttt{Embed}\left(\left[e^{H}_{1},e^{H}_{2},\cdots,e^{H}_{m}\right]\right)$, where $\mathcal{E}^{H}_{0}\in\mathbb{R}^{m\times d}$. $\mathcal{E}^{H}_{0}$ is subsequently fed into $L$ blocks $\{\phi^{(l)}\}_{i=1}^{L}$ within the transformer model. We use the superscript $(l)$ to denote the index of the block. Formally, this procedural description can be mathematically expressed as:

[TABLE]

where $e_{i}^{H,(0)}\in\mathcal{R}^{d}$ and $e_{i}\in\mathcal{R}^{d}$ denote the input path embedding and positional embedding, respectively. $\mathcal{E}^{H,(l)}=[e_{1}^{H,(l)},e_{2}^{H,(l)},\cdots,e_{m}^{H,(l)}]$. $e_{\text{cls}}^{H,(l)}$ is an additional learnable token for classification. $\phi^{(l)}$ is composed of multi-head self-attention (MHSA), a MLP layer (MLP) with layer normalization (LN) [60], and residual connection [61]. Specifically, $\phi^{(l)}$ is composed by:

[TABLE]

A single self-attention within $\texttt{MHSA}^{l}$ is calculated by softmax-weighted interactions among the input query, key, and value tokens obtained by three different learnable linear projection weights. Finally, the class prediction is achieved by a linear classification head.

3.2 Point Embedding and Tokenization

Point embedding converts the grouped raw points into a structured and representative embedding, enhancing their utilization and alignment with the input dimensionality of the foundation model, and thereby facilitating the use of its prior knowledge. We implement a lightweight network (Point_Embed) to obtain the point embedding:

[TABLE]

where Point_Embed can take various forms that incorporate local geometry aggregation operations, such as PointNet [28], PointMLP [62], PointPN [63], to name a few. The input point $x_{i}^{P}$ is from $\mathcal{P}_{g}$. We use $\mathcal{X}_{i}^{P}$ to represent the set of $k$ neighboring points $\left\{x^{P}_{i,j}\right\}_{j=1}^{k}$ around $x^{P}_{i}$ for simplicity. To seamlessly integrate with the pre-trained foundation model, the dimensionality of point embedding should align with the 2D or 1D embedding in Eq. (1). Specifically, $\hat{e}^{P}_{i}\in\mathbb{R}^{d}$. Eventually, the embedding representation of an input point cloud $\mathcal{P}$ for feeding into pre-trained foundation model is $\hat{\mathcal{E}}^{P}=\left[\hat{e}^{P}_{1},\hat{e}^{P}_{2},\cdots,\hat{e}^{P}_{N_{s}}\right]$.

The inherent unordered nature of point clouds is one of their most significant properties [28], distinguishing 3D data from pixel arrays in visual data and sequences in linguistic data. Merely aligning the dimensionality of embeddings is insufficient to fully leverage the attention-related priors of a pre-trained transformer. Based on the positional encoding in the Transformer [36], we propose the position injector. It injects sufficient positional information from the source modality into 3D tokens, enabling more effective collaboration with the frozen transformer. We use average pooling, $\texttt{avgP}:\mathbb{R}^{N_{s}\times d}\rightarrow\mathbb{R}^{1\times d}$, to obtain a global embedding $e_{g}$ that represents the centroid of the input:

[TABLE]

Then, the input point token $e^{P,(0)}_{i}$ fed into the transformer blocks is obtained by a linear combination of the relative position and the point embedding:

[TABLE]

where $a$ and $b$ are learnable parameters. They can be replaced by a 1D convolution kernel, allowing this module to be seamlessly integrated into an existing model as a standalone layer. Therefore, Eq. (6) can be changed into the following form:

[TABLE]

where Conv1D denotes 1D convolution operation, and $\texttt{Concat}\{\}$ represents the concatenation of the inputs. Since it contains only two training parameters (without using a bias term), the increase in the total number of training parameters is negligible. The structure of this position injector (PosIn) is shown in Fig. 3.

3.3 Point-Prompt Generation

Training transformer-based architectures from scratch generally requires larger datasets compared to CNN-based ones [64]. Compared to text and image data, the availability of 3D data is relatively constrained, leading to challenges such as overfitting and suboptimal utilization of the full potential of transformer-based models. This paper investigates PEFT technology to alleviate overfitting and improve model generalization for 3D models. PEFT involves the freezing of the pre-trained backbone that is previously trained on an extensive dataset, while introducing a limited number of learnable parameters to adapt to the new dataset. This new dataset can be data-rich [57, 65], few-shot [66], or long-tailed [67, 68, 69], as PEFT equips the model with knowledgeable priors.

Prompt tuning [70, 57] is one of the most effective and widely used PEFT methods. It appends trainable prompts to the tokens in Eq. (3) to fine-tune self-attention for different tasks and has been empirically validated for its effectiveness in handling both 1D linguistic and 2D visual data. We apply this technique to integrate heterogeneous prior attention with point tokens. Different from prompt tuning [70] and VPT [57], we propose a trainable prompt generator for prompt generation. Prompts for point clouds (point-prompts) should satisfy the following properties: 1) they are closely related to the input, 2) they capture the overall information of the input point clouds, and 3) they share the same dimensionality as the point embeddings. To achieve this, we adopt the same structure as token embedding and introduce a pooling operation to capture the overall features of the input, which are then used as the point-prompts. To maintain consistency between the point tokens, which capture local features, and the point-prompt, which encodes overall features, while also reducing training parameters, we make the parameters between the point embedding module and the prompt generator module shared. Consequently, the point-prompt $p_{0}$ fed into the subsequent transformer blocks is calculated as follows:

[TABLE]

where maxP and avgP refer to max pooling and average pooling, respectively. $p_{0}$ is permutation-invariant to the raw point groups, ensuring that the model remains insensitive to the order of point group arrangement (the detailed proof will be provided in Sec. 3.5). This prompt generator provides three main advantages:

1. It provides more stable global features;
2. It eliminates redundant information; and
3. The generated point-prompt preserves the geometric information of the input point clouds.

3.4 Effective Fine-Tuning of Transformer Blocks

Given a pre-trained foundation model, the generated point-prompt is incorporated into each transformer block. During fine-tuning, only the task-specific prompt generator is updated, while the transformer backbone remains fixed. The point-prompt serves two primary functions: 1) it adapts the prior self-attention mechanisms within the pre-trained transformer model; 2) it encodes the global features of the input point cloud to provide structural information - distinguishing it from existing prompt-tuning techniques. Consequently, we concatenate the generated point-prompt to each block and retain it at the output of each transformer block, preserving the original encoded point cloud structure while maintaining the interaction between the pre-trained prior and the point-prompt. The point-prompted transformer blocks are formulated as:

[TABLE]

where PosIn is calculated by Eqs. 5 and 6. $\mathcal{Z}^{l}\in\mathbb{R}^{l\times d}$ denotes the features generated by the $l$-th transformer block. The colors red and blue indicate intermediate variables that originate from trainable and frozen modules, respectively.

For the input token to the downstream head, Li et al. [69] proposed that all learnable prompts are trained on the fine-tuning dataset, thereby incorporating newly acquired information. They propose the "merge prompt" strategy, which linearly combines all the learned prompts from the final block into a class token. Inspired by this approach, in our work, both point tokens and prompts are learned from the point cloud dataset. We employ a pooling operation, following the Swin transformer [71], to integrate the newly learned knowledge into the final class token:

[TABLE]

where Pool adopts the sum of max and average pooling. APPT can be beneficial for multiple 3D downstream tasks due to its minimal training cost. Only the prompt generator, which shares weights with the point embedding module, and the task-specific head need to be trained. There are two main downstream tasks:

Classification involves labeling and categorizing the entire point cloud. The predicted logit for each class is obtained by applying the softmax function to the output of the final linear layer:

[TABLE]

where $w_{i}$ is the weight of the classification head and $C$ is the total number of classes. Eventually, the cross-entropy loss can be utilized to calculate the loss function.

Segmentation involves dividing 3D point cloud data into multiple subsets or regions with similar attributes. To achieve this, we utilize a U-Net-style architecture, where the APPT serves as the point encoder. The segmentation head concatenates the output features from the transformer blocks within the encoder, followed by deconvolutional interpolation and multiple MLP layers to enable dense prediction. Similar to the classification task, the softmax cross-entropy is used as the loss function.

3.5 Rational Analysis

In point cloud analysis, tasks such as classification and segmentation rely on the spatial distribution of points, rather than their order. We introduce the permutation-invariant [72, 73] and show this property of our method.

Definition 1**.**

(Permutation-invariant function.) For a set $S=\{s_{1},s_{2},\cdots,s_{n}\}$, a function $g:\mathbb{R}^{d_{1}\times d}\rightarrow\mathbb{R}^{d_{2}}$ is permutation-invariant iff it satisfies

[TABLE]

for any permutation $\sigma$ (any reordering of the elements).

Lemma 1**.**

The max operation, $\max:\mathbb{R}^{d}\rightarrow\mathbb{R}$, is a permutation-invariant function.

Proof.

Let $\sigma$ be an arbitrary permutation of the set $S$. By definition, $\sigma$ is a bijective function that rearranges the elements of $s$, such that $\sigma(S)=\{s_{\sigma(1)},s_{\sigma(2)},\dots,s_{\sigma(n)}\}$ for $s_{i}\in S$. Since $\max(S)$ selects the largest element in $S$, and the permutation $\sigma$ does not alter the set content, we have $\max(S)=\max(\sigma(S))$. ∎

Lemma 2**.**

The mean operation, $\texttt{mean}:\mathbb{R}^{d}\rightarrow\mathbb{R}$, is a permutation-invariant function.

Proof.

Let $\sigma$ be an arbitrary permutation of the set $S$, where $S=\{s_{1},s_{2},\dots,s_{n}\}$. The mean of the set $\sigma(S)$ is given by

[TABLE]

Since $\sigma$ is a bijective function, $\sigma(S)$ contains exactly the same elements as $S$. Furthermore, by the commutative property of addition, we can rearrange the terms in the sum without changing its value,

[TABLE]

Thus, we conclude that:

[TABLE]

∎

By Lemmas 1 and 2, we can deduce the following theorem regarding pooling operations.

Theorem 3**.**

The max pooling and average pooling across channels, $\texttt{maxP}:\mathbb{R}^{c\times d}\rightarrow\mathbb{R}^{1\times d}$ and $\texttt{avgP}:\mathbb{R}^{c\times d}\rightarrow\mathbb{R}^{1\times d}$, are both permutation-invariant functions.

The global embedding $e_{g}$ (as defined in Eq. 5 of Sec. 3.2) is utilized to determine the relative position and, therefore, must remain invariant to the ordering of point embeddings. Similarly, the point-prompt $p_{o}$ (as defined in Eq. 8 of Sec. 3.3) offers a comprehensive representation of the input, while the final class token $e_{\text{cls}}$ (as defined in Eq. 12 of Sec. 3.4) is the global feature that integrates both the input data and the prior knowledge from the foundation model. Since the order of point embeddings does not reflect the spatial relationship or structure of the input point cloud, both $p_{o}$ and $e_{\text{cls}}$ should also be unaffected by the permutation of point embeddings. Theorem 3 shows that $e_{PI}$, $p_{o}$ and $e_{\text{cls}}$ are permutation-invariant with respect to the order of point embeddings. This property enables our proposed APPT to effectively extract spatial structure and global features from point cloud data, allowing the model to better cope with noise and sampling unevenness.

4 Experiment

4.1 Datasets and Basic Settings

Datasets. We conduct object classification tasks using the widely used benchmarks, ScanObjectNN [78] and ModelNet40 [79]. ScanObjectNN is a challenging dataset with inherent scan noise and occlusion, consisting of 15,000 scanned objects across 15 distinct classes, sampled from the real world. In line with prior work, we conduct experiments on three variants: OBJ-BG, OBJ-ONLY, and PB-T50-RS. ModelNet40 contains 12,311 CAD models across 40 object categories. We follow the official data split, with 9,843 objects for training and 2,468 for evaluation, ensuring a fair comparison. For part segmentation, we utilize ShapeNetPart [80], a meticulously annotated 3D dataset derived from ShapeNet. ShapeNetPart encompasses 16 distinct shape categories, each annotated at the part level across 50 classes. Notably, each category is further delineated into 2 to 6 unique parts, providing granularity and specificity essential for detailed segmentation analysis.

Implementation Details. We follow the settings in [16] and [74], using the AdamW optimizer in combination with the Cosine annealing scheduler. The learning rate is initialized at $5\times 10^{-4}$, with a weight decay of $5\times 10^{-2}$. For the point embedding module, we explore an architecture based on Point-PN [63]. The output dimensionality of the point embedding module is set to 768 to match the input feature channels of the Transformer architecture. In comparison experiments, the ViT-Base version (ViT-B) [8] pre-trained on imageNet21K [82] is utilized as the pre-trained 2D model, which is widely adopted in previous work [16, 46]. For the 1D model, we leverage ImageBind audio encoder [83] for the audio prior and CLIP text encoder [9] for the language prior, respectively. For few-shot classification and part segmentation, we conduct experiments using the 2D pre-trained ViT-B. In the ablation study, we further investigate the impact of various pre-trained models to rigorously validate the effectiveness of our proposed APPT framework. Specifically, we employ DINOv2 [10] and DeiT [84] as alternative visual priors, and RoBERTa [85] as the linguistic prior, to assess the robustness and generalizability of our proposed APPT across different pre-trained architectures.

Comparison Methods. We compare our APPT with two primary categories of methods. The first category consists methods based on multilayer perceptron (MLP) or convolutional neural network (CNN), including foundational works such as PointNet [28], DGCNN [34], as well as more recent advancements like PointMLP [62] and Point-PN [63]. Additionally, we evaluate against 3D pre-trained models, including OcCo [13], which integrates PointNet and DGCNN. The second category comprises methods leveraging multi-head self-attention (MHSA), including the basic Transformer [36] and its adaptations for point cloud data, such as Point Cloud Transformer (PCT) [38]. Additionally, we compare our approach with methods that utilize 3D pre-trained models, such as Transformer-OcCo[13], Point-BERT [12], Point-MAE [49], Joint-MAE [74], and fine-tuned Point-BERT with Point-PEFT [48]. To ensure a comprehensive evaluation, we also include methods employing 2D pre-trained models, such as P2P [16], our conference version APF [27], and Any2Point [15]. Furthermore, we extend our comparisons to pre-trained models from other modalities, including ACT [56], ReCon [76], and Any2Point [15], which integrate audio and text data.

4.2 Comparison Results

Object Classification. Table I presents a comparative analysis of the APPT classification performance compared to the existing methods in the ScanObjectNN and ModelNet40 datasets. From the experimental results, the following observations can be drawn:

1. The integration of pre-trained models, irrespective of modality, consistently enhances model performance, albeit with varying degrees of improvement across methods. For example, incorporating the 3D pre-trained model Transformer-OcCo improves performance by 1.6% on ScanObjectNN and 0.7% on ModelNet40, demonstrating the effectiveness of leveraging 3D priors. In contrast, Joint-MAE achieves more substantial improvements of 8.9% and 2.6% on the respective datasets. These results underscore the necessity for developing more effective strategies to better harness prior knowledge and maximize performance gains.
2. APPT consistently outperforms existing SOTA methods by a large margin, particularly on the challenging real-world dataset, ScanObjectNN. For example, on the most challenging split, PB-T50-RS, the recent method Any2Point, which also employs Point-PN for point cloud tokenization, achieves accuracies of 87.7% with the visual pre-trained model and 91.9% with the textual pre-trained model, improving 0.7% and 4.8%, respectively, over Point-PN. In comparison, APPT achieves accuracies of 92.6% with the visual pre-trained model and 91.4% with the textual pre-trained model, delivering remarkable gains of 5.5% and 4.3%, respectively, over Point-PN. On ModelNet40, APPT outperforms Any2Point across all corresponding pre-trained modalities and surpasses other SOTA competitors. For instance, APPT with the textual pre-trained model achieves an accuracy of 95.1%, exceeding Any2Point by 0.8% and ReCon, which integrates 3D+2D+1D pre-trained modalities, by 1.7%. Overall, our method demonstrates superior performance.

Few-shot Classification. To demonstrate the generalization capability of the proposed APPT, we conduct experiments under few-shot settings, following the common protocol established in [12, 74]. The ‘$N$-way, $K$-shot’ configuration is a conventional setup, wherein $N$ classes are randomly selected, with each class containing $K$ training samples and 20 testing samples. Each experimental setting was repeated 10 times, and the results are reported as the mean performance accompanied by the standard deviation. The results are summarized in Table II. Compared to both 2D and 3D pre-trained models, APPT exhibits superior generalization ability in few-shot learning. For instance, APPT achieves notable improvements of 3.0%, 3.2%, 3.3%, and 2.9% over the 3D pre-trained model Transformer-OcCo in four distinct settings. Furthermore, even compared to recently proposed SOTA methods such as Point-MAE, Joint-MAE, and our conference version APF, APPT consistently outperforms these approaches in terms of both accuracy and stability. The only exception occurs in the 10-way 20-shot setting, where APPT marginally underperforms compared to APT. These results underscore the robustness and efficacy of the proposed APPT framework in few-shot learning tasks.

Part Segmentation. In alignment with established methodologies [28, 49, 74], we sample 2,048 points from each input instance and adopt the same segmentation head as utilized in Point-MAE [49] and Joint-MAE [74]. The corresponding results are detailed in Table III. Although APPT may not outperform SOTA methods across all evaluation metrics, it exhibits competitive overall performance. Notably, APPT outperforms both P2P and our conference version APF, both of which leverage image priors, underscoring its enhanced capability in integrating multimodal information. Furthermore, although APPT marginally lags behind Joint-MAE in terms of $\text{mIoU}_{C}$ and $\text{mIoU}_{I}$, it is crucial to emphasize that Joint-MAE necessitates training from scratch, a process that demands substantially greater computational resources and training time. In contrast, APPT requires significantly lower computational overhead, making it a more efficient and practical alternative for segmentation tasks.

4.3 Further Analysis

Ablation Study of Individual Modules. To systematically assess the contribution of each module within APPT, we conduct controlled experiments, with the experimental settings and results comprehensively outlined in Table IV. The results demonstrate that each module plays a crucial role in enhancing the performance of the baseline method, which employs the point embedding (PE) module based on Point-PN. Notably, the pre-trained model on the 2D modality, along with the point-prompt tuning (PPT) and position injection (PosIn) modules yield substantial performance improvements across both datasets, highlighting their pivotal contributions to the overall effectiveness of APPT.

To further elucidate the contribution of each module, we visualize the feature distribution and the corresponding response on the original input point clouds, as shown in Figs. 4 and 5, respectively. Specifically, when the point embedding module (PE, namely Point-PN) is employed independently, the feature distribution across categories exhibits overlap, as shown in Fig. 4a. Fig. 4b illustrates the feature distribution after the PosIn module aligns with the ViT-B architecture, which is built upon the Point-PN framework and leverages the 2D pre-trained model. Meanwhile, Fig. 4c demonstrates the effectiveness of PPT module, utilizing the same PE module and 2D pre-trained model. When combined with the visualizations in Fig. 5, it becomes evident that PPT and PosIn focus on distinct regions of the object; however, both modules emphasize the object structure, thereby enhancing the separability of the learned representations. This complementary focus underscores the synergistic contribution of PPT and PosIn to the overall performance of the framework. Finally, Fig. 4d demonstrates the combined effect of APPT. The third row of Fig. 5 reveals that APPT captures a relatively complete and coherent overall structure of the object. This observation helps explain why APPT achieves significant improvement in classification but performs slightly inferior to Joint-MAE in segmentation, as APPT encoder prioritizes the global structure of the input over fine-grained local details. Intuitively, the feature distribution boundaries obtained by APPT are more distinct, with a notable enhancement in feature separation.

The Impact of Different Foundation Models. We compare the performance of APPT across different pre-trained foundation models on ScanObjectNN PB-T50-RS dataset. The corresponding results are summarized in Table V, which also includes comparisons with other methods using the same pre-trained models. Except when leveraging textual pre-trained knowledge, APPT consistently outperforms all other methods with the same pre-trained foundation models. For example, with the 2D prior, APPT outperforms APF (pre-trained on ViT-B) by 4.8% and Any2Point (pre-trained on DINOv2) by 4.9%. Although APPT slightly lags behind Any2Point when using textual pre-trained knowledge on ScanObjectNN (91.4% vs. 91.7% and 87.3% vs. 89.7%), it significantly outperforms Any2Point when leveraging the 1D audio prior (88.9% vs. 87.0%). On ModelNet40 (see Table I), APPT also achieves superior performance compared to Any2Point with text prior (95.1% vs. 94.3%). Furthermore, experiments with other pre-trained base models, such as DeiT [84] (visual prior) and ImageBind [83] (audio prior), show that APPT consistently outperforms the baseline method (88.9% and 88.9% vs. 87.1%) by a clear margin. Additionally, Table VI provides a comparison of APPT with baseline methods, demonstrating its performance improvement with the use of multiple modalities. These results underscore the robustness and versatility of APPT across diverse pre-trained models and modalities.

Comparison of Trainable Parameters. Table VII provides a comparison of APPT with SOTA methods based on pre-trained foundation models, with a focus on the number of trainable parameters. In contrast to P2P and Any2Point, our method introduces more parameters during point token embedding, yet yields a notable performance improvement. On the other hand, APPT significantly reduces the number of training parameters compared to Point-MAE and Point-BERT, while simultaneously delivering notable performance gains, attributed to its efficient fine-tuning strategy. Additionally, compared to APF, APPT further reduces the number of trainable parameters and improves model performance through the implementation of a shared weights strategy. Improving the efficiency of training parameters will remain a primary focus of our future research.

5 Conclusion

This paper has proposed an innovative PEFT architecture, APPT, designed to effectively leverage diverse pre-trained foundation models for 3D understanding tasks. It leverages the rich semantic information embedded in large pre-trained models to efficiently enhance 3D understanding tasks, thereby addressing the challenges of data scarcity and overfitting often faced by 3D pre-trained models. APPT departs from the existing projection-based method by adopting a point embedding module to maximize the retention of high-dimensional structural information from point clouds. A permutation-invariant feature is then utilized to determine the relative positions of point embeddings, enhancing the understanding of point cloud structures while effectively leveraging the priors embedded in heterogeneous pre-trained models. The attention mechanism of the pre-trained large model is adapted through point-prompts generated by a shared weights prompt generator, ensuring efficient and scalable integration of pre-trained knowledge. Extensive experiments have demonstrated that APPT exhibits strong generalization capabilities across various heterogeneous foundation models, achieving significant performance improvements in 3D understanding tasks.