RoFt-Mol: Benchmarking Robust Fine-Tuning with Molecular Graph Foundation Models

TL;DR

This paper benchmarks various fine-tuning methods for molecular graph foundation models, identifies their strengths and weaknesses, and introduces ROFT-MOL, a new robust fine-tuning approach that improves performance across tasks.

Contribution

It provides a comprehensive benchmark of eight fine-tuning methods for MGFMs and proposes ROFT-MOL, a novel method combining weight interpolation and ensemble techniques.

Findings

ROFT-MOL outperforms existing fine-tuning methods in diverse tasks.

Benchmark reveals strengths and limitations of different fine-tuning mechanisms.

Diverse evaluation settings highlight the importance of robust fine-tuning strategies.

Abstract

In the era of foundation models, fine-tuning pre-trained models for specific downstream tasks has become crucial. This drives the need for robust fine-tuning methods to address challenges such as model overfitting and sparse labeling. Molecular graph foundation models (MGFMs) face unique difficulties that complicate fine-tuning. These models are limited by smaller pre-training datasets and more severe data scarcity for downstream tasks, both of which require enhanced model generalization. Moreover, MGFMs must accommodate diverse objectives, including both regression and classification tasks. To better understand and improve fine-tuning techniques under these conditions, we classify eight fine-tuning methods into three mechanisms: weight-based, representation-based, and partial fine-tuning. We benchmark these methods on downstream regression and classification tasks across supervised and self-supervised pre-trained models in diverse labeling settings. This extensive evaluation provides valuable insights and informs the design of a refined robust fine-tuning method, ROFT-MOL. This approach combines the strengths of simple post-hoc weight interpolation with more complex weight ensemble fine-tuning methods, delivering improved performance across both task types while maintaining the ease of use inherent in post-hoc weight interpolation.

Tables

Table 1. Table 1: A summary of evaluated pre-trained models and their corresponding result tables for reference. “ Clf ” and “ Rgs ” represent classification and regression tasks, respectively, while “ Non ” and “ Few ” denote Non-Fewshot and Fewshot settings.

Objectives	Models	Reference Tables of Experimental Results
		Clf-Non	Clf-Few	Rgs-Non	Rgs-Few
Self-Supervised	Mole-BERT	2	6	3	7
	GraphMAE	12	14	13	15
	MoleculeSTM	8	10	9	11
Supervised	Graphium-Toy	2	6	3	7
	Graphium-Large	8	10	9	11
	GraphGPS	12	14	13	15

Table 2. Table 17: DWiSE-FT performance on 2 Regression datasets (RMSE metrics) and 2 Classification datasets (AUC) in the Fewshot setting with 50 samples, evaluated across dataset splits ( Scaffold, Size ) given GraphGPS model. Avg-R denote the average rank. Standard deviations across five replicates are shown in parentheses. We bold and underline the best and second-best performances in each scenario.

Method	Few-Shot 50 (Scaffold Split)						Few-Shot 50 (Size Split)
	BACE (AUC)	SIDER (AUC)	Avg AUC	ESOL (RMSE)	LIPO (RMSE)	Avg R	BACE (AUC)	SIDER (AUC)	Avg AUC	ESOL (RMSE)	LIPO (RMSE)	Avg R
WiSE-FT	$54.67\pm 0.12$	$83.73\pm 0.00$	$69.23$	$1.020\pm 0.045$	$1.259\pm 0.027$	$3$	$53.03\pm 2.64$	$83.77\pm 0.34$	$68.40$	$1.045\pm 0.054$	$1.230\pm 0.015$	$4$
L2-SP	$59.52\pm 3.80$	$84.94\pm 0.16$	$72.23$	$0.897\pm 0.058$	$1.196\pm 0.030$	$2$	$61.93\pm 3.45$	$85.12\pm 0.28$	$73.53$	$0.851\pm 0.036$	$1.194\pm 0.015$	$2.5$
TOP	$64.90\pm 1.55$	$85.12\pm 0.23$	$75.01$	$0.859\pm 0.065$	$1.196\pm 0.030$	$1.5$	$61.93\pm 3.45$	$86.48\pm 0.70$	$74.21$	$0.844\pm 0.007$	$1.180\pm 0.014$	$1.5$
DWiSE-FT	$64.82\pm 1.53$	$85.23\pm 0.02$	$75.03$	$0.859\pm 0.071$	$1.190\pm 0.016$	$\mathrm{𝟏}$	$61.46\pm 0.57$	$85.39\pm 0.23$	$73.43$	$0.868\pm 0.041$	$1.167\pm 0.016$	$2$

Table 3. Table 19: LoRA Performance under few-shot and non-fewshot settings across classification and regression datasets with pretrained model GraphGPS.

Scaffold Split	clintox	bbbp	bace	hiv	sider	esol	lipo	malaria	cep
fewshot 50	$97.77\pm 0.21$	$60.15\pm 3.83$	$57.99\pm 3.48$	$67.95\pm 4.21$	$83.02\pm 0.24$	$0.796\pm 0.032$	$1.232\pm 0.039$	$1.188\pm 0.008$	$2.081\pm 0.082$
fewshot 500	$99.79\pm 0.02$	$100.00\pm 0.00$	$99.99\pm 0.02$	$80.19\pm 1.52$	$92.16\pm 0.25$	$0.354\pm 0.009$	$0.260\pm 0.010$	$0.872\pm 0.006$	$1.569\pm 0.040$
non-fewshot	$99.80\pm 0.02$	$99.84\pm 0.06$	$99.54\pm 0.21$	$94.98\pm 0.66$	$90.84\pm 0.11$	$0.375\pm 0.033$	$0.318\pm 0.009$	$0.737\pm 0.015$	$0.632\pm 0.017$

Equations

$$\underset{\{\mathbf{\theta },\mathbf{\varphi }\}}{\min }\sum _{i=1}^N\mathcal{L}(g_{\mathbf{\varphi }}\circ f_{\mathbf{\theta }}({\mathcal{G}}_i),y_i),$$

$${\mathbf{Q}}^T{\tilde{\mathbf{\theta }}}_{\text{ft}}=(\mathbf{\Lambda }+\delta \mathbf{I}{)}^{-1}\mathbf{\Lambda }{\mathbf{Q}}^T{\mathbf{\theta }}_{\text{ft}}^{\ast }+\delta (\mathbf{\Lambda }+\delta \mathbf{I}{)}^{-1}{\mathbf{Q}}^T{\mathbf{\theta }}_{\text{pre}}.$$

$${\mathbf{\theta }}^{\left[i\right]}=\left(1-{\alpha }_i\right)\cdot {\mathbf{\theta }}_{\mathrm{pre}}^{\left[i\right]}+{\alpha }_i\cdot {\mathbf{\theta }}_{\mathrm{ft}}^{\left[i\right]}$$

$${\mathbf{Q}}^T{\tilde{\mathbf{\theta }}}_{\text{ft}}=(\mathbf{\Lambda }+\delta \mathbf{I}{)}^{-1}\mathbf{\Lambda }{\mathbf{Q}}^T{\mathbf{\theta }}_{\text{ft}}^{\ast }+\delta (\mathbf{\Lambda }+\delta \mathbf{I}{)}^{-1}{\mathbf{Q}}^T{\mathbf{\theta }}_{\text{pre}}.$$

$$\mathcal{L}(\mathbf{\theta })$$

$$=\mathcal{L}({\mathbf{\theta }}_{\text{ft}}^{\ast })+\frac{1}{2}(\mathbf{\theta }-{\mathbf{\theta }}_{\text{ft}}^{\ast }{)}^T\mathbf{H}(\mathbf{\theta }-{\mathbf{\theta }}_{\text{ft}}^{\ast })$$

$$\tilde{\mathcal{L}}(\mathbf{\theta })=\mathcal{L}({\mathbf{\theta }}_{\text{ft}}^{\ast })+\frac{1}{2}(\mathbf{\theta }-{\mathbf{\theta }}_{\text{ft}}^{\ast }{)}^T\mathbf{H}(\mathbf{\theta }-{\mathbf{\theta }}_{\text{ft}}^{\ast })+\delta \Vert {\mathbf{\theta }}_{\text{ft}}-{\mathbf{\theta }}_{\text{pre}}{\Vert }_2^2$$

$$\mathbf{H}({\tilde{\mathbf{\theta }}}_{\text{ft}}-{\mathbf{\theta }}_{\text{ft}}^{\ast })+\delta ({\tilde{\mathbf{\theta }}}_{\text{ft}}-{\mathbf{\theta }}_{\text{pre}})=0$$

$$(\mathbf{H}+\delta \mathbf{I}){\tilde{\mathbf{\theta }}}_{\text{ft}}=\mathbf{H}{\mathbf{\theta }}_{\text{ft}}^{\ast }+\delta {\mathbf{\theta }}_{\text{pre}}$$

$${\tilde{\mathbf{\theta }}}_{\text{ft}}=(\mathbf{H}+\delta \mathbf{I}{)}^{-1}(\mathbf{H}{\mathbf{\theta }}_{\text{ft}}^{\ast }+\delta {\mathbf{\theta }}_{\text{pre}})$$

$${\tilde{\mathbf{\theta }}}_{\text{ft}}=(\mathbf{Q}\mathbf{\Lambda }{\mathbf{Q}}^T+\delta \mathbf{I}{)}^{-1}(\mathbf{Q}\mathbf{\Lambda }{\mathbf{Q}}^T{\mathbf{\theta }}_{\text{ft}}^{\ast }+\delta {\mathbf{\theta }}_{\text{pre}})$$

$${\tilde{\mathbf{\theta }}}_{\text{ft}}=(\mathbf{Q}(\mathbf{\Lambda }+\delta \mathbf{I}){\mathbf{Q}}^T{)}^{-1}(\mathbf{Q}\mathbf{\Lambda }{\mathbf{Q}}^T{\mathbf{\theta }}_{\text{ft}}^{\ast }+\delta {\mathbf{\theta }}_{\text{pre}})$$

$${\mathbf{Q}}^T{\tilde{\mathbf{\theta }}}_{\text{ft}}=(\mathbf{\Lambda }+\delta \mathbf{I}{)}^{-1}\mathbf{\Lambda }{\mathbf{Q}}^T{\mathbf{\theta }}_{\text{ft}}^{\ast }+\delta (\mathbf{\Lambda }+\delta \mathbf{I}{)}^{-1}{\mathbf{Q}}^T{\mathbf{\theta }}_{\text{pre}}$$

Full text

1 Introduction

In recent years, foundation models [bommasani2021opportunities, zhou2023comprehensive] have achieved success in learning high-quality, general-purpose representations of images and text through pre-training on diverse datasets [radford2021learning, kirillov2023segment, ramesh2022hierarchical, touvron2023llama, bubeck2023sparks, zhao2023survey]. To adapt these pre-trained models for downstream applications, additional training on task-specific data, known as fine-tuning, is often required. However, vanilla fine-tuning frequently encounters challenges, including model overfitting [howard2018universal, lirethinking, kornblith2019better], catastrophic forgetting of pre-trained knowledge [leesurgical, lidelta, xuhong2018explicit, lubana2022quadratic], and distribution shifts between fine-tuned and test samples, which can lead to negative transfer [wang2019characterizing, chen2019catastrophic]. These challenges highlight the need for robust fine-tuning strategies [shen2021partial, wortsman2022robust, kumar2022fine, shu2023clipood, andreassenevolution, kirichenkolast].

Recently, the advantages of foundation models have been extended to various scientific applications [golling2024masked, leung2024towards, nguyen2023climax]. Among these, molecular graph foundation models (MGFMs) have gained significant attention for their promising potential in biochemistry [hu2020strategies, hou2022graphmae, xiamole, suresh2021adversarial, shoghimolecules, beainitowards, zheng2023towards, ross2022large, rong2020self, maoposition]. While MGFMs exhibit scaling behaviors similar to foundation models in other domains [chen2024uncovering], they face unique challenges related to data and tasks.

A primary challenge stems from the significantly smaller pre-training datasets in this domain, typically consisting of at most $O(100M)$ molecular samples, compared to the billions of samples used in other domains [sun2022does]. This limitation restricts the parameter scale of MGFMs ($O(100M)$ parameters) and their generalization capacity [wang2024evaluating, akhondzadeh2023probing]. Furthermore, downstream tasks in this domain often involve limited data for fine-tuning, with datasets containing only tens or a few hundred labeled samples [wijaya2024two], exacerbating the difficulty of achieving robust model generalization. In addition to data constraints, many downstream tasks, such as molecular property prediction, are regression-based [wu2018moleculenet, hou2022accurate]. These tasks require models to capture fine-grained numerical patterns, which presents a distinct requirement compared to the coarse-grained feature reliance typical in classification tasks in CV and NLP. These factors collectively highlight the need for a careful examination of fine-tuning strategies for MGFMs and their appropriate improvement.

To answer this question, we introduce RoFt-Mol, a benchmark that evaluates existing fine-tuning methods across diverse molecular property prediction tasks. To explore factors influencing the fine-tuning (FT) performance of MGFMs, we categorize 8 FT methods into 3 distinct mechanisms: 1) weight-based FT, which ensembles the weights from both pre-trained and fine-tuned models, 2) representation-based FT, which regularizes the proximity between pre-trained and fine-tuned latent data representations, and 3) partial FT, which optimizes only a subset of the pre-trained model’s weights while keeping the rest frozen. To derive generalizable insights into how different fine-tuning mechanisms interact with pre-training strategies and downstream task types, we evaluate six diverse pre-trained models, spanning self-supervised and supervised learning, with pure graph-based, graph transformer based and multi-modal models in varying scales, then evaluate on a broad set of molecular property prediction tasks, including 8 classification and 4 regression tasks. To simulate the challenges encountered during the fine-tuning stages of MGFMs, we further consider the few-shot and out-of-distribution settings. Drawing from the broad range of pre-trained models and downstream tasks, we indeed find that the choice of best fine-tuning mechanism is highly determined by the pre-training objective and the downstream task type. We summarize high-level insights as follows, with further detailed results presented in Sec. 4. The bold text within brackets indicates the corresponding support in the experiment sections for clear cross-referencing:

• •

Impact from Supervised vs. Self-supervised pre-trained models: Supervised pre-training learns domain-specific information with task supervision, while self-supervised pre-training captures general-purpose knowledge through training on generic synthetic tasks. We observe that, in few shot fine-tuning, supervised pre-training generally yields better fine-tuning performance than self-supervised pre-training even when the pre-training tasks do not align well with the fine-tuning tasks. In contrast, for non-few-shot settings, supervised pre-training performs better only when the supervised pre-training tasks closely align with the downstream tasks [Q2].

• •

Impact from Classification vs. Regression tasks: Regression tasks need more precise numerical labels and finer molecule modeling. Therefore, MGFMs face less risk of overfitting in regression tasks compared to classification tasks, particularly in the few-shot setting [Q1].

• •

Correspondence with different fine-tuning methods: For self-supervised pre-trained models, weight-based fine-tuning often results in better performance by effectively integrating general knowledge from pre-training with task-specific knowledge from fine-tuning [Finding 1]. On the other hand, partial fine-tuning typically leads to underfitted molecular representations in few-shot fine-tuning, particularly for regression tasks [Finding 2]. For supervised pre-trained models, representation-based fine-tuning performs well due to the preservation of domain-relevant pre-trained representations [Finding 3].

Based on the findings, we argue that the first step in selecting or designing an effective fine-tuning strategy is to consider the pre-training strategies. Then after finding the suitable fine-tuning mechanisms, we need to take the type of downstream tasks into account. For instance, weight-based fine-tuning methods generally work the best under self-supervised pre-trained model, while simple post-hoc weight interpolation between pre-trained and fine-tuned model weights (WiSE-FT) performs well for classification tasks but struggles with regression tasks. In contrast, a more complex weight ensemble approach ($L^{2}$-SP) achieves better performance in regression tasks, though it comes with the cost of increased tuning complexity. Therefore, inspired by the rule, we propose a new method, DWiSE-FT that achieves strong performance for both regression and classification tasks as a weight-based solution for self-supervised pre-trained model. DWiSE-FT combines the strengths of WiSE-FT and $L^{2}$-SP, providing strong performance for both task types while maintaining the plug-and-play ease of post-hoc interpolation. The success of DWiSE-FT showcases that our benchmark identifies valuable insights in improving fine-tuning strategies given distinct MGFMs.

2 Finetuning Methods for Evaluation

In this section, we briefly introduce representative methodologies used in pre-training and fine-tuning for MGFMs.

Self-supervised Pre-training strategies have been proven to be effective in generating transferable molecular representations for downstream tasks [zhao2024survey]. In a high level, they can be divided into reconstruction methods and contrastive methods. The generative-based strategies adopt mask-based graph reconstruction by utilizing graph autoencoders [hou2022graphmae, tan2023s2gae, wang2017mgae, pan2018adversarially], context predictions [hu2020strategies, rong2020self] and generative language model pre-training [hu2020gpt, zhang2021motif]. On the other hand, contrastive-based methods aim for maximizing the similarity between perturbed instance pairs [velivckovic2018deep, suresh2021adversarial, you2020graph, xia2023mole, wang2022molecular, zhu2022unified, you2021graph, qiu2020gcc, li2022let, xu2021self]. Moreover, the advancement of language models has prompted numerous studies to employ multi-modal frameworks. These approaches harness language models to enhance molecular understanding through techniques such as cross-modal contrastive learning and alignment [su2022molecular, liu2023multi, seidl2023enhancing, liu2023molca].

In this work, we select GraphMAE [hou2022graphmae] as the representative of the recontruction-based pre-trained model, which focuses on masked feature reconstruction with scaled cosine error that enabled robust training. Regarding the contrastive pre-trained model, we choose Mole-BERT [xia2023mole] that combines the node-level masked atom modeling to predict the masked atom tokens and the graph-level contrastive learning through triplet loss and contrastive loss. Lastly, we choose MoleculeSTM [liu2023multi] as the representative of multi-modal molecule structure-text model that jointly learning molecules’ chemical structures and textual descriptions via a contrastive learning strategy.

Supervised Pre-training. Recently, to leverage more diverse datasets and tasks, researchers started exploring the ability of supervised pre-training with multi-task learning for molecular representations [gasteigergemnet, shoghimolecules, beainitowards]. We adopt pre-trained models trained on multi-task labeled samples in a supervised manner from the Graphium library [beainitowards]. In addition to the GNN-based backbone, more expressive architectures like Graph Transformer [ying2021transformers, rampavsek2022recipe, mialon2021graphit] have been proposed and can be used as the pre-trained backbone with supervised labels, which we adopt GraphGPS [rampavsek2022recipe] as a representative.

Fine-tuning’s overall goal is to adapt the pre-trained model to downstream applications. Specifically, given a pre-trained GNN encoder $f_{\bm{\theta}}$ with parameters $\bm{\theta}$ initialized from the pretrained parameters $\bm{\theta}_{\text{pre}}$, fine-tuning optimizes the encoder $f_{\bm{\theta}}$ and an additional prediction head $g_{\bm{\phi}}$ with parameters $\bm{\phi}$ over downstream molecules $\{(\mathcal{G}_{i},y_{i})\}_{i=1}^{N}$. The vanilla version, full-FT, optimizes the entire model weights following:

[TABLE]

where $\bm{\theta}$ is initialized as $\bm{\theta}_{\text{pre}}$ and $\mathcal{L}$ denotes the loss function for prediction tasks. As discussed, there are advanced fine-tuning strategies proposed on top of the full-FT framework. As shown in Fig. 1, we group them into three categories based on their mechanisms and benchmark representative methods for each category. More FT methods that fall into each category or others will be discussed in Appendix C.

$\bullet$ Partial FT strategies only optimizes partial weights of the pre-trained model, i.e., a subset of weights within $\{\bm{\theta},\bm{\phi}\}$ will be updated following the same objective as Eq. 1. Linear Probing (LP) only trains the additional prediction head $g$ during the FT. Surgical FT [leesurgical] updates only partial layers within the encoder. For instance, we can update the weights for $k$-th layer of the GNN encoder as $\min_{\{[\bm{\theta}]_{k},\bm{\phi}\}}\sum_{i=1}^{N}\mathcal{L}(g_{\bm{\phi}}\circ f_{\bm{\theta}}(\mathcal{G}_{i}),y_{i})$, where $k$ is the hyperparameter that can be tuned. LP-FT [kumar2022fine] aims to address the issue of pre-trained feature distortion during the full-FT process. It first performs the LP step to the prediction head $g_{\bm{\phi}}$ while keeping the encoder $f_{\bm{\theta}}$ with fixed pre-trained parameters $\bm{\theta}_{\text{pre}}$, followed by applying full-FT with the tuned prediction head.

$\bullet$ Weight-based FT strategies mainly update the entire model weights through combining pre-trained model weights and fine-tuned model weights. WiSE-FT [wortsman2022robust] linearly interpolates between pre-training parameters $\bm{\theta}_{\text{pre}}$ and fine-tuning parameters $\bm{\theta}_{\text{ft}}$ using a mixing coefficient $\alpha$, to get the interpolated GNN $f_{\bm{\theta}_{\text{int}}}$ with weights $\bm{\theta}_{\text{int}}=(1-\alpha)\cdot\bm{\theta}_{\text{pre}}+\alpha\cdot\bm{\theta}_{\text{ft}}$. We first perform full-FT to obtain the adapted encoder $f_{\bm{\theta}_{\text{ft}}}$ and classifier $g_{\bm{\phi}}$, then apply post-hoc weight ensembling to get $f_{\bm{\theta}_{\text{int}}}$, with final predictions given by $g_{\bm{\phi}}\circ f_{\bm{\theta}_{\text{int}}}(\mathcal{G}_{i})$. $\alpha$, as a hyperparameter, controls the weight ensemble. $L^{2}$-SP [xuhong2018explicit] regularizes the fine-tuning model weights $\bm{\theta}$ closer to the pre-trained weights $\bm{\theta}_{\text{pre}}$ by $\Omega(\bm{\theta},\bm{\phi})=\frac{\delta}{2}\lVert\bm{\theta}-\bm{\theta}_{\text{pre}}\rVert^{2}_{2}$. We optimize for $\bm{\theta}$ and $\bm{\phi}$ by combining the prediction loss from Eq. 1 and $\Omega(\bm{\theta},\bm{\phi})$ with tunable trade-off coefficient $\delta$.

$\bullet$ Representation-based FT methods mainly regulate the latent representation space during FT. Feature-map [lidelta] adds distance regularization between the latent representations of pre-trained and fine-tuned models to the Full-FT loss. The regularization is defined as $\Omega(\bm{\theta})=\delta\sum_{i=1}^{N}\frac{1}{2}\lVert f_{\bm{\theta}}(\mathcal{G}_{i})-f_{\bm{\theta}_{\text{pre}}}(\mathcal{G}_{i})\rVert^{2}_{2}$, where $\delta$ controls the regularization strength. BSS [chen2019catastrophic] aims at resolving the negative transfer issue through eliminating the spectral components corresponding to small singular values that are less transferable. The regularization is done as $\Omega({\bm{F}})=\delta\sum_{i=1}^{k}\sigma^{2}_{-i}$, where ${\bm{F}}=[f_{\bm{\theta}}(\mathcal{G}_{0}),\cdots,f_{\bm{\theta}}(\mathcal{G}_{b})]$ is the feature matrix of a batch of graphs and $\sigma_{-i}$ are the $i$-th smallest singular values obtained from the SVD of ${\bm{F}}$. We can tune $k$ and $\delta$ to determine the number of singular values to penalize and the degree of penalty.

3 Experimental Settings in the Benchmark

In this section, we briefly introduce the experimental settings in this work. More detailed experimental settings can be found in Appendix F.

Foundation Models. For self-supervised pre-training, we adopt three open-source pre-trained checkpoints: Mole-BERT, GraphMAE, and MoleculeSTM. For supervised pre-training, we use models from the Graphium [beainitowards] library, which get pre-trained on the Toymix and Largemix datasets provided in this library. To differentiate between them, we refer to these models as Graphium-Toy and Graphium-Large. For larger graph transformer based model, we adopt the pre-trained checkpoint of GraphGPS [rampavsek2022recipe] pre-trained on the PCQM4MV2 [hu2ogb]. For details of datasets used in pre-training are in Appendix D. Furthermore, we include the traditional baseline XGBoost [chen2016xgboost] for Fewshot scenarios to better compare with the foundation model in Appendix G.2.

Downstream Datasets. We use 8 classification and 4 regression datasets for downstream task evaluation. Detailed statistics and references for these tasks are in Appendix E.

$\dagger$ Classification. The BBBP dataset measures if a molecule will penetrate blood-brain barrier. The Tox21, ToxCast, and ClinTox datasets are related to toxicity qualitative measurements. The Sider dataset stores qualitative results of different types of adverse drug reactions. The MUV dataset is specifically designed for validation of virtual screening techniques. The HIV dataset provides qualitative activity results of the molecular ability to inhibit HIV replication. The BACE dataset contains qualitative binding results for a set of inhibitors of human $\beta$-secretase 1 (BACE-1).

$\dagger$ Regression. Esol is a dataset which measures aqueous solubility of molecules. The Lipo dataset measures the octanol-water partition coefficient. Cep is a subset of the Havard Clean Energy Project (CEP), which estimates the organic photovoltaic efficiency. Malaria measures the drug efficacy against the parasite that causes malaria.

Dataset Splits. For each downstream dataset, we experiment with random, scaffold, and size splits to create the Train/Val/Test subsets. Specifically, the random splitting shuffles the data, maintaining the Train/Val/Test sets as in-distribution (ID). The other two splitting methods simulate out-of-distribution (OOD) challenges in real-world applications. For scaffold splitting, we follow prior works [ramsundar2019deep], ensuring structural differences in molecular scaffolds across splits. Size splitting, following zougess, arranges molecules in ascending order by size, evaluating model generalization across different molecule sizes.

Size of fine-tuning samples. In practice, molecular property prediction tasks can have very limited experimentally-validated data, e.g., with less than 100 samples [wijaya2024two]. Thus, we consider both Non-Fewshot and Fewshot settings to better simulate the label scarcity issue. In the Non-Fewshot setting, we use all available samples from the splitted train set. In the Fewshot settings, we sample subsets of 50, 100, and 500 molecules from the Train set for fine-tuning, while keeping the Val/Test sets unchanged to ensure a fair comparison. Note that we exclude MUV, Tox21, and ToxCast datasets for the Fewshot settings, as we cannot randomly select training samples while ensuring that all tasks have a specified number of labels simultaneously, due to the severe label scarcity issues in these datasets.

Evaluation Metrics. We use AUC to evaluate the performance for classification datasets and RMSE for regression datasets. We report the model performance over 5 random seeds and the test performance are reported based on the best validation performance. The AVG, AVG-F, AVG-R denote the average metrics, average metrics without max and min values, and average rank over all the datasets for each evaluated method, respectively.

4 Results and Analysis

This section mainly analyzes the experimental results from Mole-BERT and Graphium-Toy models as representatives of self-supervised and supervised pre-training. Table 1 is a summary of all pre-trained models we test on and their corresponding result tables for reference. Since we observe similar trends from pre-trained models of the same category, we will refer to them in our result analysis and compare over different pre-trained models in Sec. 4.3. Due to limited space, more findings with different fine-tuning methods and pre-trained models comparison can be found in Appendix G.

4.1 Self-supervised Pre-trained Models

Q1: How does self-supervised pre-training influence downstream prediction tasks?

(1a) Regression tasks require more task-specific knowledge from downstream fine-tuning compared to classification tasks.

When checking the few-shot results in Fig. 2a and 2c, full fine-tuning ranks the highest for regression tasks but only achieves mid-tier performance for classification tasks. This disparity likely arises from the distinct nature of these tasks. Classification tasks typically require coarser-grained features, as exemplified by the Tox21 dataset. In this case, determining toxicity may largely rely on recognizing certain functional groups, such as toxicophores or structural alerts [singh2016toxicophore]. In contrast, regression tasks demand finer-grained features. For example, predicting precise solubility involves factors such as partial charge distribution, conformational flexibility, and hydrogen bond patterns, among others [faller2007computational]. Consequently, models fine-tuned for regression tasks must acquire more downstream knowledge during the fine-tuning process and are generally less prone to overfitting compared to those used for classification tasks.

(1b) Molecular representations learned from self-supervised pre-training are not informative enough for downstream tasks.

As shown in Tables 2 and 3, LP is consistently the worst performing method for self-supervised pre-trained models across all data splits, even under the few-shot fine-tuning in Fig. 2a and 2c. Furthermore, this behavior is widely observed across all tested self-supervised models as GraphMAE and MoleculeSTM, which contrasts the observations in CV where LP demonstrates robust OOD performance by preserving high quality and generalizable features from pre-trained embeddings [wortsman2022robust, kumar2022fine]. We attribute this to the misalignment between general-purpose representations produced by self-supervised pre-training and the features required by the specific molecular tasks. Consequently, relying solely on tuning the classifier $g_{\bm{\phi}}$ is insufficient to extract meaningful predictions from these non-informative representations.

Below, we summarize insightful findings from the performance of different fine-tuning strategies.

$\bullet$ Finding 1. Under few-shot fine-tuning, weight-based fine-tuning strategies stand out with WiSE-FT for classification tasks and $L^{2}$-SP for regression tasks.

Among various fine-tuning methods, weight-based approaches consistently outperform others across a wide range of experiments, regardless of the few-shot sample sizes (Fig. 2a and 2c). Self-supervised models are known to capture general-purpose knowledge for substructure discovery[wang2024evaluating]. During fine-tuning, combining pre-trained and fine-tuned weights proves effective in extracting molecular patterns relevant to downstream tasks. Notably, WiSE-FT demonstrates superior performance on classification datasets, whereas $L^{2}$-SP excels in regression tasks. This finding is also supported by MoleculeSTM in table 11 where $L^{2}$-SP remains as top method under all few-shot regression tasks and WiSE-FT excels under Fewshot-50 classification. Essentially, WiSE-FT applies a straightforward post-hoc linear interpolation between pre-trained and fine-tuned models, governed by a single coefficient. In contrast, $L^{2}$-SP implicitly determines the weight combination through the training loss [lubana2022quadratic, xuhong2018explicit], aligning with statement (1a) that regression tasks typically demand more nuanced modeling.

$\bullet$ Finding 2. Partial FT results in underfitted molecular representations under Fewshot settings, which is more severe for regression tasks compared to classification.

For the non-few-shot fine-tuning (Tables 2 and 3), surgical FT and LP-FT improve over full FT in both classification and regression tasks. However, in few-shot fine-tuning, both methods rank as the worst methods. This is likely because partial fine-tuning underfits and bias towards the the limited samples. This issue is more pronounced in regression tasks.

4.2 Supervised Pre-trained Models

Q2: How does supervised pre-training influence downstream tasks?

We first discuss the task similarity between the datasets used in the pre-training and downstream fine-tuning process. As introduced in Appendix. D, the ToyMix dataset used for supervised pre-training contains QM9, Tox21 and Zinc12K. The predictions from QM9 are not directly related to our downstream tasks, but may involve indirect correlations, as the quantum chemical properties provided by QM9 are highly valuable for characterizing molecular features. Tox21 is an overlapping dataset that also exists as one of the downstream datasets. Its tasks in predicting qualitative toxicity measurements are highly related to the downstream ClinTox and ToxCast datasets, and also correlate to the Sider dataset which contains evaluation in drug side effects. Lastly, Zinc12K, which is to predict the constrained solubility, is relevant to the Esol and Lipo datasets that involve solubility predictions. Other downstream tasks do not share the same tasks with pre-training directly. Then we observe the following rules.

(2a) Under few-shot fine-tuning, supervised pre-training models generally yield higher fine-tuning performance compared to self-supervised pre-training, regardless of the pre-training and fine-tuning task correlations.

Supervised pre-training brings more benefits to downstream tasks than self-supervised pre-training in few-shot situations when checking Tables 6 and 7. Besides, the benefits are less relevant to the task similarity in contrast to the non-few-shot cases. For example, the improvements are also observed in HIV and Cep datasets even their tasks do not share with pre-training tasks directly. This implies that learned domain-specific knowledge still offer better insights than generic knowledge when fine-tuning guidance is minimal.

(2b) Under non-few-shot fine-tuning, fine-tuning performance given supervised pre-training outperforms self-supervised pre-training when its objectives closely align with downstream tasks, while task misalignment may harm performance.

From Tables 2 and 3, we observe consistent fine-tuning performance improvements over self-supervised pre-training on highly task-correlated downstream datasets including ClinTox, Esol, Lipo and Tox21. Even when pre-training involves regression tasks and downstream tasks are classification, performance gains occur if the physical meanings align. For datasets that do not directly share tasks with pre-training, we observe mixed performance on Sider, Malaria, and Cep datasets, and even worse performance on HIV and MUV datasets. This observation contrasts to few-shot cases in (2a), which entails that downstream task specific knowledge can be learned given sufficient guidance on top of generic knowledge from self-supervised pre-training.

Below are some detailed findings with different fine-tuning methods given supervised pre-training.

$\bullet$ Finding 3. Fine-tuning strategies that regularizes towards pre-trained molecular representations rank top, while weight-based methods are suboptimal.

From non-few-shot (Tables 2 and 3) and few-shot fine-tuning (Figs. 2b and 2d) in both supervised models with ToyMix and LargeMix, surgical FT and Feature-map tend to be the top-ranking methods. However, best performing weight-based methods for self-supervised pre-training, only show mediocre performance here. This can also be observed in the larger-scale GraphGPS model as discussed in Appendix G.1. In addition, the other representation-based method BSS shows limited performance compared to Feature-map, which directly regularizes the distance to pre-trained representations. These observations suggest that given the task alignment between supervised pre-training and downstream fine-tuning, pre-trained representations tend to contain transferable features for downstream tasks. Consequently, controlling the degree to preserve pre-trained representations is the key to downstream fine-tuning performance.

4.3 Discussions over Pre-trained Models

Our extensive evaluation shows that the ranking of fine-tuning techniques remains consistent across pre-trained models within the same category, either supervised or self-supervised, regardless of model architecture, scale, or pre-training dataset. This suggests that our guidance for selecting fine-tuning methods based on the pre-training paradigm is broadly applicable and generalizable across diverse model designs. For instance, self-supervised models such as Mole-BERT and MoleculeSTM tend to benefit more from weight-based fine-tuning, while supervised models like Graphium and GraphGPS perform better with feature-based approaches.

5 Methodology Exploration

Based on findings in Sec. 4, we observe that weight-based fine-tuning generally performs well under self-supervised pre-training. However, the top strategy varies: WiSE-FT excels in classification tasks, while $L^{2}$-SP is more effective for regression tasks. This motivates us to further explore the connections and trade-offs between these methods to identify potential improvements. In this section, we introduce DWiSE-FT, an extension of the weight ensemble method unifying the strengths from WiSE-FT and $L^{2}$-SP. DWiSE-FT demonstrates top-ranking results through efficient post-processing that better suits the practical fine-tuning needs.

5.1 Motivation

As introduced in Sec. 2, WiSE-FT adopts the post-hoc linear interpolation between the pre-trained and fine-tuned model weights as $(1-\alpha)\cdot\bm{\theta}_{\text{pre}}+\alpha\cdot\bm{\theta}_{\text{ft}}$. Although $L^{2}$-SP does not explicitly have weight interpolation in the form, the optimal weight $\tilde{\bm{\theta}}_{\text{ft}}$ from the weight-regularized loss $\tilde{\mathcal{L}}(\bm{\theta})$ is indeed the linear interpolation of the optimal model from full FT $\bm{\theta}^{*}_{\text{ft}}$ and the pre-trained model $\bm{\theta}_{\text{pre}}$.

Proposition 1**.**

Given $\tilde{\mathcal{L}}(\bm{\theta})=\mathcal{L}(\bm{\theta})+\frac{\delta}{2}\|{\bm{\theta}}-\bm{\theta}_{\text{pre}}\|^{2}_{2}$, we define the optimal weights as $\tilde{\bm{\theta}}_{\text{ft}}=\mathrm{argmin}_{\bm{\theta}}\tilde{\mathcal{L}}(\bm{\theta})$ and $\bm{\theta}^{*}_{\text{ft}}=\mathrm{argmin}_{\bm{\theta}}{\mathcal{L}}(\bm{\theta})$.

[TABLE]

where ${\bm{H}}$ is the hessian matrix of $\mathcal{L}$ evaluated at $\bm{\theta}^{*}_{\text{ft}}$ and ${\bm{H}}={\bm{Q}}{\bm{\Lambda}}{\bm{Q}}^{T}$.

Namely, $L^{2}$-SP can be seen as a more tailored weight ensemble method, employing variable mixing coefficients for different weights. This approach balances the influence of the prediction loss and the degree of weight regularization, unlike the fixed interpolation controlled by $\alpha$ across all weights in WiSE-FT. By accounting for subtle differences in loss values, $L^{2}$-SP is better suited for regression tasks, which are more sensitive to numerical variations.

While $L^{2}$-SP excels on regression datasets, its regularization coefficient is less interpretable and necessitates retraining when experimenting with different values. In contrast, WiSE-FT offers a simpler and more flexible approach, performing post-hoc interpolation without additional training once the model is fine-tuned once. Furthermore, the mixing coefficient $\alpha$ is both easy to adjust and straightforward to interpret. Therefore, our goal is to find a method that benefits from both WiSE-FT and $L^{2}$-SP to accommodate regression and classification tasks at the same time.

5.2 Algorithm

We propose DWiSE-FT that shares the framework of using the $\alpha$ to control the weight ensemble between the pre-trained model and fine-tuned model. The key idea, inspired by Eq. 4 is to enable different $\alpha$ values when ensembling the weights for different encoder layers as shown in Fig. 1. Given the pre-trained model with parameters $\bm{\theta}_{\text{pre}}$ and model after full fine-tuning with parameters $\bm{\theta}_{\text{ft}}$, The interpolated model has weights $\bm{\theta}^{\left[i\right]}$ with mixing coefficient $\alpha_{i}$ for the $i$-th layer as:

[TABLE]

This approach naturally incorporates the characteristics of $L^{2}$-SP and even surgical FT: The weight ensemble in DWiSE-FT offers the flexibility through varying mixing layer-wise coefficients between the pre-trained and fine-tuned models, addressing the limitations of WiSE-FT. Additionally, we enable the selection of $\bm{\alpha}$ through optimization via validation loss gradient inspired by the Gradient-based Neural Architecture Search (NAS) [dong2019searching].

5.3 Experiment results

Regarding the classification datasets, DWiSE-FT should have the performance at least as good as WiSE-FT since WiSE-FT is the special case of DWiSE-FT with one fixed mixing coefficient. We evaluate DWiSE-FT to see how it improves upon WiSE-FT and matches the superior performance of $L^{2}$-SP for regression tasks under few-shot fine-tuning. Please note that, due to space constraints, we only present the experiments for few-shot fine-tuning with 50 and 100 samples in the main text. The complete table is available in Appendix E, Table LABEL:table:new_method_appendix. In Table LABEL:table:new_method_reg, we compare DWiSE-FT’s performance against WiSE-FT, $L^{2}$-SP, and the best-performing method in each setting. Specifically, we find that DWiSE-FT consistently outperforms WiSE-FT. Furthermore, DWiSE-FT often surpasses $L^{2}$-SP or at least maintains comparable results in most scenarios. Additionally, in some cases, DWiSE-FT even exceeds the performance of the best-performing methods. Therefore, DWiSE-FT can be a great candidate for fine-tuning on regression datasets in practice since it guarantees top performance with easier usage.

6 Conclusion

This work benchmarks totally 8 fine-tuning methods, categorizing them into three groups, and evaluate them across 12 downstream datasets under 36 different experimental settings covering 3 dataset splits, 4 training sample sizes, and 6 molecular pre-trained models. The design of these settings reflects practical demands of molecular representation fine-tuning under 1) diversified foundation model with both supervised and self-supervised pre-training, 2) wide range of downstream tasks in both classification and regression that has not been widely studied by previous literature and 3) scarcely labeled molecules for fine-tuning. The study analyzes what is needed when facing classification vs. regression tasks and when given supervised vs. self-supervised pre-training. Then, we provide insights in best performing fine-tuning methods accordingly under aforementioned scenarios. Additionally, we propose an extended fine-tuning method DWiSE-FT, driven by our observations, that maintains top-ranking results through a more efficient and automated design for certain fine-tuning scenarios. This highlights the value of our benchmark in offering valuable insights for both fine-tuning methodology design and practical guidance in molecular representation learning.

7 Acknowledgments

S. Liu, D. Zou, N. Shoghi and P. Li are partially supported by NSF awards PHY-2117997, IIS-2435957, IIS-2428777 and IIS-2239565; the JP-Morgan Chase Faculty Award, the NVIDIA Academic Grant Program and the IDEaS Cyberinfrastructure Awards.

NeurIPS Paper Checklist

Claims 2. Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope? 3. Answer: [Yes] 4. Justification: The main claims are justified by the experimental results and discussion in sec 4. Also, we refer the claims to the later findings in introduction. 5. Guidelines:

• •

The answer NA means that the abstract and introduction do not include the claims made in the paper.

• •

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

• •

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

• •

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 6. 2.

Limitations 7. Question: Does the paper discuss the limitations of the work performed by the authors? 8. Answer: [Yes] 9. Justification: We include a section in appendix discussing our limitations and future works. 10. Guidelines:

• •

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

• •

The authors are encouraged to create a separate "Limitations" section in their paper.

• •

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

• •

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

• •

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

• •

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

• •

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

• •

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren’t acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 11. 3.

Theory assumptions and proofs 12. Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? 13. Answer: [Yes] 14. Justification: For the proposition 1 included in the paper, we have the complete proof in the appendix. 15. Guidelines:

• •

The answer NA means that the paper does not include theoretical results.

• •

All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.

• •

All assumptions should be clearly stated or referenced in the statement of any theorems.

• •

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

• •

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

• •

Theorems and Lemmas that the proof relies upon should be properly referenced. 16. 4.

Experimental result reproducibility 17. Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)? 18. Answer: [Yes] 19. Justification: We detail the experimental settings in the sec 3 and more hyperparameter tuning and dataset details in Appendix. 20. Guidelines:

• •

The answer NA means that the paper does not include experiments.

• •

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

• •

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

• •

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

• •

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

1. (a)

If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. 2. (b)

If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully. 3. (c)

If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset). 4. (d)

We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 21. 5.

Open access to data and code 22. Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? 23. Answer: [Yes] 24. Justification: The access to datasets and codes are provided and we include the detailed settings in the paper main text and appendix. 25. Guidelines:

• •

The answer NA means that paper does not include experiments requiring code.

• •

Please see the NeurIPS code and data submission guidelines (https://nips.cc/public/guides/CodeSubmissionPolicy) for more details.

• •

While we encourage the release of code and data, we understand that this might not be possible, so “No” is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

• •

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https://nips.cc/public/guides/CodeSubmissionPolicy) for more details.

• •

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

• •

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

• •

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

• •

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 26. 6.

Experimental setting/details 27. Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results? 28. Answer: [Yes] 29. Justification: We clarify the dataset splits, hyperparameters and evaluation in the main text and appendix. 30. Guidelines:

• •

The answer NA means that the paper does not include experiments.

• •

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

• •

The full details can be provided either with the code, in appendix, or as supplemental material. 31. 7.

Experiment statistical significance 32. Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? 33. Answer: [Yes] 34. Justification: We include the standard deviation for all the experiments. 35. Guidelines:

• •

The answer NA means that the paper does not include experiments.

• •

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

• •

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

• •

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

• •

The assumptions made should be given (e.g., Normally distributed errors).

• •

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

• •

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

• •

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

• •

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 36. 8.

Experiments compute resources 37. Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments? 38. Answer: [Yes] 39. Justification: We include the computing resources in the appendix. 40. Guidelines:

• •

The answer NA means that the paper does not include experiments.

• •

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

• •

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

• •

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn’t make it into the paper). 41. 9.

Code of ethics 42. Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 43. Answer: [Yes] 44. Justification: We follow the code of ethics. 45. Guidelines:

• •

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

• •

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

• •

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 46. 10.

Broader impacts 47. Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? 48. Answer: [Yes] 49. Justification: We include the broader impact discussion in appendix. 50. Guidelines:

• •

The answer NA means that there is no societal impact of the work performed.

• •

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

• •

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

• •

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

• •

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

• •

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 51. 11.

Safeguards 52. Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? 53. Answer: [N/A] 54. Justification: Our paper poses no such risks. 55. Guidelines:

• •

The answer NA means that the paper poses no such risks.

• •

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

• •

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

• •

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 56. 12.

Licenses for existing assets 57. Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? 58. Answer: [Yes] 59. Justification: We cite the sources of the datasets that are used. 60. Guidelines:

• •

The answer NA means that the paper does not use existing assets.

• •

The authors should cite the original paper that produced the code package or dataset.

• •

The authors should state which version of the asset is used and, if possible, include a URL.

• •

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

• •

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

• •

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

• •

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

• •

If this information is not available online, the authors are encouraged to reach out to the asset’s creators. 61. 13.

New assets 62. Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? 63. Answer: [Yes] 64. Justification: The new assets introduced in the paper are well documented. 65. Guidelines:

• •

The answer NA means that the paper does not release new assets.

• •

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

• •

The paper should discuss whether and how consent was obtained from people whose asset is used.

• •

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 66. 14.

Crowdsourcing and research with human subjects 67. Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)? 68. Answer: [N/A] 69. Justification: The paper does not involve crowdsourcing nor research with human subjects. 70. Guidelines:

• •

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

• •

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

• •

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 71. 15.

Institutional review board (IRB) approvals or equivalent for research with human subjects 72. Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained? 73. Answer: [N/A] 74. Justification: The paper does not involve crowdsourcing nor research with human subjects. 75. Guidelines:

• •

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

• •

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

• •

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

• •

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 76. 16.

Declaration of LLM usage 77. Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required. 78. Answer: [N/A] 79. Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components. 80. Guidelines:

• •

The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

• •

Please refer to our LLM policy (https://neurips.cc/Conferences/2025/LLM) for what should or should not be described.

Appendix A Proof of proposition 1

Proposition 2**.**

Given $\tilde{\mathcal{L}}(\bm{\theta})=\mathcal{L}(\bm{\theta})+\frac{\delta}{2}\|{\bm{\theta}}-\bm{\theta}_{\text{pre}}\|^{2}_{2}$, we define the optimal weights as $\tilde{\bm{\theta}}_{\text{ft}}=\mathrm{argmin}_{\bm{\theta}}\tilde{\mathcal{L}}(\bm{\theta})$ and $\bm{\theta}^{*}_{\text{ft}}=\mathrm{argmin}_{\bm{\theta}}{\mathcal{L}}(\bm{\theta})$.

[TABLE]

where ${\bm{H}}$ is the hessian matrix of $\mathcal{L}$ evaluated at $\bm{\theta}^{*}_{\text{ft}}$ and ${\bm{H}}={\bm{Q}}{\bm{\Lambda}}{\bm{Q}}^{T}$.

Proof.

Based on the quadratic approximation, we can approximate $\mathcal{L}(\bm{\theta})$ as follows:

[TABLE]

since $\mathcal{L}^{\prime}(\bm{\theta}^{*}_{\text{ft}})=0$ as $\bm{\theta}^{*}_{\text{ft}}$ is the minimum. Then, we add the weight regularization term, such that

[TABLE]

Then, we solve for $\tilde{\bm{\theta}}_{\text{ft}}$ by setting $\nabla\tilde{\mathcal{L}}(\bm{\theta})=0$

[TABLE]

∎

Appendix B Limitations and Future Works

We acknowledge certain limitations in this current work and highlight potential improvements for future research. Firstly, this study primarily focuses on the property prediction tasks of small molecules using 2D-graph based foundation models. Exploring a broader array of foundation models across a wider range of applications–such as covering more areas like DNA, proteins, and materials, addressing various scientific tasks like linker design and chemical reactions, and incorporating diverse data formats like 3D geometric data–is highly worthwhile. Secondly, although we attempt to include many representative fine-tuning methods from various categories in this study, additional fine-tuning methods from different categories, as discussed in Appendix C, deserve investigation. For instance, future research could explore whether graph-specific fine-tuning methods offer additional benefits over non-graph fine-tuning approaches across various settings we design. Thirdly, the method DWiSE-FT introduced here is an extension and combination of existing methods directly motivated by our benchmark findings for specific fine-tuning scenarios. Future work may involve more thorough exploration into fine-tuning methodology design inspired by our current findings, and aiming to develop approaches effective across a broader range of fine-tuning scenarios.

Regarding the broader impact, we recognize our work can be beneficial to the drug discovery and material science, but people should be aware of the misuse of molecular property prediction tasks to harmful chemical production.

Appendix C Additional Discussions of Related Works

In this section, we additionally discuss more related works about fine-tuning (FT) techniques. Designing advanced fine-tuning strategies first gained attention in the computer vision (CV) and natural language processing (NLP) domains, leading to the development of various research directions. We categorize the mainstream approaches into the following groups.

Partial model FT. Numerous studies demonstrate that freezing certain parameters while fine-tuning only specific components of the pre-trained model can help mitigate overfitting during the fine-tuning process [kirkpatrick2017overcoming, lee2019would, ramasesh2020anatomy, eastwood2021source, evci2022head2toe, cohen2022my]. Specifically, Linear Probing (LP) only trains the additional prediction head during FT. Surgical FT [leesurgical] selectively fine-tunes a subset of layers based on the specific mechanism of distribution shifts. Partial FT is similar to the concept of parameter efficient fine-tuning methods like LoRA [hu2022lora], Prefix tuning [li2021prefix] and IA3 [liu2022few]. We also include an additional study on LoRA performance in App. G.3.

Weight-based FT strategies mainly control the model weights during the FT. Specifically, WiSE-FT [wortsman2022robust], grounded on the linear mode connectivity [frankle2020linear], linearly interpolates between pre-training parameters and fine-tuning parameters by a mixing coefficient. $L^{2}$-SP [xuhong2018explicit] regularizes the fine-tuning model weights using $L^{2}$ distance to constrain the parameters around pre-trained ones. REGSL [li2021improved] further introduces a layer-wise parameter regularization, where the constraint strength gradually reduces from the top to bottom layers. MARS-SP [gouk2020distance] adopts the projected gradient method (PGM) to constrain the fine-tuning model weights within a small sphere centered on the pre-trained ones. More recently, TPGM [tian2023trainable] further incorporates trainable weight projection radii constraint for each layer, inspired by MARS-SP, to support layer-wise regularization optimization.

Representation-based FT methods mainly regulate the latent representation space during FT. Feature-map [lidelta] adds distance regularization between the latent representations of pre-trained and fine-tuned models to the Full-FT loss. DELTA [li2019delta] specifically constrains feature maps with the pre-trained activations selected by channel-wise attention. BSS [chen2019catastrophic] penalizes the spectral components corresponding to small singular values that are less transferable to prevent negative transfer. li2020representation proposes to transfer representations by encouraging small deviations from the reference one through an regularizer based on optimal transport. Inspired by this, GTOT-Tuning [zhang2022fine] presents optimal transport-based fine-tuning framework. LP-FT [kumar2022fine] first performs LP to prediction head while keeping the pre-trained encoder fixed, followed by applying full-FT with the tuned prediction head.

Architecture Refinement. Besides the weight and representation based FT, StochNorm [kou2020stochastic] refactors the widely used Batch Normalization (BN) module and proposes Stochastic Normalization, to transfer more pre-trained knowledge during the fine-tuning process and mitigate over-fitting.

Contrastive-based FT. As discussed in Sec. 2, contrastive-based strategies have been widely demonstrated to be effective in the pre-training stage. There are other works which explore its effectiveness in the fine-tuning process. gunel2020supervised, Bi-tuning [zhong2020bi], Core-tuning [zhang2021unleashing] and COIN [pan2023improving] introduce supervised contrastive learning [khosla2020supervised] to better leverage the label information in the target datasets with more discriminative representations as a result. More recently, FLYP [goyal2023finetune] shows that simply finetuning a classifier via the same contrastive loss as pre-training leads to superior performance in finetuning image-text models. oh2024geodesic fine-tunes the model with contrastive loss on additional hard negative samples, which are generated by geodesic multi-modal Mixup, for robust fine-tuning in multi-modal models.

Graph-specific fine-tuning techniques. Apart from the CV and NLP domains, several fine-tuning techniques specifically designed for the Graph-ML domain have recently been proposed. GTOT-Tuning [zhang2022fine] achieves efficient knowledge transfer from the pre-trained models by an optimal transport-based FT framework. Bridge-Tune [huang2024measuring] introduces an intermediate step that bridges pre-training and downstream tasks by considering the task similarity between them. G-tuning [sun2024fine] tunes the pre-trained GNN so that it can reconstruct the generative patterns (graphons) of the downstream graphs. li2024adaptergnn leverages expressive adapters for GNNs, to boost adaptation to the downstream tasks.

Appendix D Pre-training Datasets Detail

For self-supervised pre-training, Mole-BERT and GraphMAE are pre-trained over 2M molecules sampled from the ZINC15 database [sterling2015zinc], following previous works [hu2019strategies]. MoleculeSTM is initially trained on PubChemSTM, a large multimodal dataset comprising over 280,000 chemical structure–text pairs contructed from the PubChem database [kim2021pubchem].

For supervised pre-training, we use the models from the Graphium [beainitowards] library, which get pre-trained on the Toymix and Largemix datasets provided in this library. The ToyMix dataset [beainitowards], totally 2.61M graph-level data points, contains QM9 [ramakrishnan2014quantum], Tox21 [wu2018moleculenet] and Zinc12K [dwivedi2023benchmarking]. Specifically, QM9 consists of 19 graph-level quantum properties associated to an energy-minimized 3D conformation of the molecules. Zinc12K is to predict the constrained solubility which is the term $\rm logP-SA-cycle$ (octanol-water partition coefficients, logP, penalized by the synthetic accessibility score, SA, and number of long cycles, cycle). The Largemix dataset, totally 343.4M graph-level data points and 197.7M node-level data points, contains four different datasets with tasks taken from quantum chemistry (PCQM4M_G25_N4), bio-assays (PCBA1328) and transcriptomics (L1000 VCAP and MCF7). Specifically, L1000 VCAP and MCF7 are from the LINCS L1000 database [subramanian2017next], which is generated using high-throughput transcriptomics. VCAP and MCF7 are, respectively, prostate cancer and human breast cancer cell lines. The PCQM4M_G25_N4 dataset is sourced from the PubChemQC project [nakata2017pubchemqc] that computed DFT properties on the energy-minimized conformation of 3.8M small molecules from PubChem. The PCBA1328 dataset, originally sourced from wang2017pubchem, comprises 1,328 assays and 1.56M molecules and contains information about a molecule’s biological activity across various assay settings. The pretraining dataset for GraphGPS is PCQM4Mv2, which is a large-scale molecular dataset containing 3.75M graphs curated from PubChemQC. The task is to regress the HOMO-LUMO gap, a quantum physical property originally calculated using Density Functional Theory.

Appendix E Dataset Statistics

The statistics and references of the downstream datasets included in this work are shown in Table 5.

Appendix F Details of Experimental Implementation

Pre-training Implementations. For self-supervised pre-training, we use the open-source pre-trained checkpoints of Mole-BERT222https://github.com/junxia97/Mole-BERT and GraphMAE333https://github.com/THUDM/GraphMAE. For supervised pre-training, we follow the same training pipeline as proposed in the Graphium444https://github.com/datamol-io/graphium. We drop out the task head MLPs used for supervised pre-training during the downstream fine-tuning process, keeping only the graph encoder component. Note that we keep the architecture of the GNN encoder and the graph pooling strategy the same across the three pre-training models. Specifically, we use a 5-layer Graph Isomorphism Networks (GINs) with 300 hidden dimension and mean pooling as the readout function.

Fine-tuning Implementations. We keep the same training configurations across all the downstream datasets, pre-training models, and fine-tuning strategies, following hu2020strategies. Specifically, for each distinct setting, we fine-tune the pre-training models with 5 random seeds (0-4). We use a batch size of 32 and a dropout rate of 0.5. For each dataset, We train models for 100 epochs and report the test performance when the optimal validation performance is achieved.

Hyperparameter Tuning. We set learning rate to be 0.001 for all the methods and train for 100 epochs. Below is the detailed sets of hyperparameters we tuned for each fine-tuning strategy.

• •

Surgical FT: We tune $k$ as which layer in GNN encoder to be updated from $\{0,1,2,3,4\}$ since our backbone architecture is a 5-layer GIN.

• •

WiSE-FT: We tune the mixing coefficient $\alpha$ from $\{0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8,0.9\}$ to control the weight ensemble from pre-trained model and fine-tuned model. A larger $\alpha$ indicates the weights are adopted more from the fine-tuned model.

• •

$L^{2}$-SP/ BSS/ Feature-map: For these three methods that involve an additional regularization term in the loss, we tune the regularization coefficient $\delta$ from $\{1,0.1,0.01,0.001,0.0001\}$ to control the degree of regularization. For BSS, we follow the original paper and set $k$ to be 1 meaning that we are regularizing the smallest singular value.

• •

LP-FT: We train the LP step before full fine-tuning for 100 epochs and then use the updated prediction head as initilization for the full-FT afterwards for 100 epochs. The training all use the default learning rate 0.001.

• •

Full FT/ LP: There is no additional hyperparameter tuning, where we use the default fine-tuning setting.

• •

DWiSE-FT: We tune the initialization of $\alpha_{i}$ for each layer $i$, where we use the same value to initialize for all layers from $\{0.9,0.7,0.5\}$ and the learning rate for validation loss descent from $\{0.001,0.005,0.01\}$. We tune $\bm{\alpha}$ over validation sets over 200 epochs.

Indeed, from the DWiSE-FT experiments with different starting points of mixing coefficients, the variance of final results is small since it will converge towards the optimal value of mixing coefficients regardless of the initial starting point given a reasonable training time.

Computing Resources The experiments are run on NVIDIA RTX A6000 with 48G memory.

Appendix G Further Result Discussions

G.1 Comparisons over pre-trained models

We mainly select the pretrained models based on their pre-training objective divided as supervised and self-supervised learning as discussed in 2. Then, among each category of pretrained models, we diversify with different architecture, model size and detailed training objective or pretraining dataset to discover the effect to the downstream finetuning method selection.

In the following, we will briefly discuss some more results that are not included in the main text with more pretrained models we tried. Detailed tables can be found in Appendix H

In general, we found the trend discussed in the main text about the difference of supervised pretrained model and self-supervised pretrained model hold in most cases. Especially, how they prefer over the representation based finetuning techniques or the weight based finetuning techniques remain consistent. However, some small variations may happen regarding the model size and architecture. For instance, for smaller model like 5 layer base GIN model, it is less likely to overfit on fewshot dataset compared to the larger scale graph transformer model. Also, the model expressiveness and capability will vary with different model scale. Therefore, we can compare the rank of different finetuning methods under pretrained models with the same scale, while it is not directly comparable if the model scale is significantly different.

For instance, both the Graphium model and the GraphGPS demonstrate superior performance from the representation based method like feature-map and BSS compared to other techniques. However, in contrast to the Graphium-Toy model results in the main text that feature-map perform better than BSS especially under the very few shot scenarios. In the GraphGPS results, we find that feature-map tend to be better with more finetuning samples and BSS tends to be better than feature-map in the fewshot cases. This might be due to the variation in the model size that leads to more overfitting, where BSS regularize over noisy feature space through penalizing smaller eigenvalues can be more crucial in reducing overfitting compared to feature-map. Also, we experience a change in pretrained dataset compared to the ToyMix and LargeMix in the Graphium model, where the PCQM4Mv2 is less diversed. This might also cause the degraded performance of feature-map under GraphGPS with fewshot scenario since the learned representation from pretraining might not directly fit the downstream task. When there are more samples available, there might be a larger overlap with the learned representation space. Furthermore, we also observe the worse performance of LP and LP-FT under the larger model which coincides with findings in the main text from Graphium models.

The conclusions presented in Section 4 generalize well to models pre-trained on large-scale datasets, such as GraphGPS (pre-trained on PCQM4MV2) and Graphium-Large (trained on the LargeMix dataset containing hundreds of millions of labeled molecular graphs). In Section 4.3 and Appendix G.1, we analyze the consistency of trends across all six pre-trained models in our benchmark. Below, we summarize key observations that hold true for models trained on large-scale data:

• •

Supervised pre-training on large datasets leads to stronger downstream performance, particularly in few-shot settings. This aligns with our main conclusion in Section 4.2 (Q2), where we compare supervised and self-supervised pre-training. Models like GraphGPS and Graphium-Large consistently outperform self-supervised models such as Mole-BERT and GraphMAE under the same fine-tuning protocols.

• •

Representation-based fine-tuning methods (e.g., Feature-map and BSS) remain top-performing strategies for supervised pre-trained models on large-scale datasets, consistent with Finding 3. This trend holds across both classification and regression tasks, and across different dataset splits.

• •

Partial fine-tuning methods (e.g., LP and Surgical-FT) continue to underperform in few-shot settings. This observation supports Finding 2, and reflects their tendency to underfit in data-scarce regimes, even when the underlying pre-trained model is strong.

G.2 Comparisons over traditional method

To further understand the effect from foundation model pre-training and fine-tuning process, we include the XGBoost algorithm as a representative for the traditional method. Specifically, we tested the XGBoost algorithm under the Fewshot setting with 50, 100 and 500 samples to see whether it can surpass the performance of foundation model when the training data is scarce. The featurizer being used for the XGBoost model is the Extended Connectivity Circular Fingerprints adopted from the MoleculeNet paper. Then, we keep the exact same splits with the other experiments under random, scaffold and size split. From the result in table 18, we can conclude that foundation model result (e.g.) from Mole-BERT surpass the performance in XGBoost on almost all the settings. This indicates the benefit from the pretraining and finetuning framework and the value of our work in selecting the best finetuning technique given different pretraining situation.

G.3 Study on parameter efficient fine-tuning methods

As an additional study over parameter efficient fine-tuning method, we incorporate the LoRA [hu2022lora] results for GraphGPS under the scaffold split across three regimes: Fewshot-50, Fewshot-500, and non-fewshot. The results are shown in the table LABEL:table:lora.

Across both classification and regression tasks, LoRA falls short of full fine-tuning in roughly two-thirds of cases, with the gap widening for regression when more samples are available. This pattern is unsurprising since more challenging tasks and larger downstream datasets generally require updating a greater number of parameters. In the instances where LoRA does outperform full-FT, its results typically lie between standard full fine-tuning and the strongest fine-tuning baselines. Notably, under the Fewshot-50 regression setting, LoRA occasionally matches or even exceeds the best benchmarked fine-tuning methods, highlighting its potential in extremely low-data scenarios.

G.4 Additional study of DWiSE-FT on other pretrained models

We additionally test DWiSE-FT on other pretrained model like GraphGPS. As shown in Table LABEL:table:dwise-gps, we report the results of fewshot fine-tuning with 50 samples under scaffold and size splits. These results show that DWiSE-FT not only significantly improves over WiSE-FT and $L^{2}$-SP, but also matches or exceeds the best-performing method (TOP) in some cases. This demonstrates that DWiSE-FT remains effective even under supervised pre-training, including on models like GraphGPS where traditional weight-based methods struggle.

G.5 Additional findings

$\bullet$ Finding 4. LP with pre-trained molecular representations from supervised pre-training surpasses full FT under few-shot fine-tuning, except for size splits.

For few-shot fine-tuning with 50 and 100 samples (c.f., Fig. 2b and 2d), LP surpasses full FT in random and scaffold splits, differing from self-supervised pre-training discussed in (1a). This again supports the claim that directly adopting molecular representations from supervised pre-training retain useful knowledge for downstream tasks. But interestingly, this does not hold for size splits. We believe it is due to the susceptibility of graph level tasks under size shift, as noted in prior OOD studies [zougess]. Namely, the prediction head tends to overfit to the mapping from representations to output labels with molecules in a specific range of sizes, and thus cannot generalize to OOD molecules of different sizes.

$\bullet$ Finding 5. Regulating feature representations brings significant benefits under few-shot fine-tuning but has only a marginal impact in non-few-shot fine-tuning.

Representation-based methods incorporates additional representation regularization in addition to full FT. BSS aims to eliminate noisy or non-transferable dimensions by regularizing small singular values of representations and Feature-map enforces a close distance of the fine-tuned representations to the pre-trained representations. Since the baseline full FT performs well under non-few-shot settings (c.f., Tables 2 and 3), and pre-trained molecular representations are unsatisfying as discussed in Q1, having fine-tuned representations to unsatisfying pre-trained representations does not lead to any benefits. While under few-shot fine-tuning, representation regularization prevents overfitting with limited samples on top of full FT to some extend.

Appendix H Additional Experimental Results

In this section, we present complementary baseline results over all pretrained models that are not shown in the main text due to space limit. Table 1 is a summary of all pre-trained models we test on and their corresponding result tables for reference. Also, a complete table including all few-shot fine-tuning results for DWiSE-FT (including Fewshot 500 case omitted in the main text) are in Table LABEL:table:new_method_appendix.