# RoFt-Mol: Benchmarking Robust Fine-Tuning with Molecular Graph Foundation Models

## TL;DR

This paper benchmarks various fine-tuning methods for molecular graph foundation models, identifies their strengths and weaknesses, and introduces ROFT-MOL, a new robust fine-tuning approach that improves performance across tasks.

## Contribution

It provides a comprehensive benchmark of eight fine-tuning methods for MGFMs and proposes ROFT-MOL, a novel method combining weight interpolation and ensemble techniques.

## Key findings

- ROFT-MOL outperforms existing fine-tuning methods in diverse tasks.
- Benchmark reveals strengths and limitations of different fine-tuning mechanisms.
- Diverse evaluation settings highlight the importance of robust fine-tuning strategies.

## Abstract

In the era of foundation models, fine-tuning pre-trained models for specific downstream tasks has become crucial. This drives the need for robust fine-tuning methods to address challenges such as model overfitting and sparse labeling. Molecular graph foundation models (MGFMs) face unique difficulties that complicate fine-tuning. These models are limited by smaller pre-training datasets and more severe data scarcity for downstream tasks, both of which require enhanced model generalization. Moreover, MGFMs must accommodate diverse objectives, including both regression and classification tasks. To better understand and improve fine-tuning techniques under these conditions, we classify eight fine-tuning methods into three mechanisms: weight-based, representation-based, and partial fine-tuning. We benchmark these methods on downstream regression and classification tasks across supervised and self-supervised pre-trained models in diverse labeling settings. This extensive evaluation provides valuable insights and informs the design of a refined robust fine-tuning method, ROFT-MOL. This approach combines the strengths of simple post-hoc weight interpolation with more complex weight ensemble fine-tuning methods, delivering improved performance across both task types while maintaining the ease of use inherent in post-hoc weight interpolation.