Meta-Learning GNN Initializations for Low-Resource Molecular Property Prediction

TL;DR

This paper evaluates meta-learning methods, specifically MAML and its variants, for initializing graph neural networks to improve low-resource molecular property prediction, showing significant performance gains over traditional pre-training.

Contribution

It demonstrates that meta-initializations learned via MAML significantly enhance GNN performance in low-resource drug discovery tasks, outperforming multi-task pre-training on most benchmarks.

Findings

Meta-initializations outperform pre-training on 16/20 in-distribution tasks.

Meta-initializations outperform on all out-of-distribution tasks.

Consistent best performance across various fine-tuning data sizes.

Abstract

Building in silico models to predict chemical properties and activities is a crucial step in drug discovery. However, limited labeled data often hinders the application of deep learning in this setting. Meanwhile advances in meta-learning have enabled state-of-the-art performances in few-shot learning benchmarks, naturally prompting the question: Can meta-learning improve deep learning performance in low-resource drug discovery projects? In this work, we assess the transferability of graph neural networks initializations learned by the Model-Agnostic Meta-Learning (MAML) algorithm - and its variants FO-MAML and ANIL - for chemical properties and activities tasks. Using the ChEMBL20 dataset to emulate low-resource settings, our benchmark shows that meta-initializations perform comparably to or outperform multi-task pre-training baselines on 16 out of 20 in-distribution tasks and on all out-of-distribution tasks, providing an average improvement in AUPRC of 11.2% and 26.9% respectively. Finally, we observe that meta-initializations consistently result in the best performing models across fine-tuning sets with $k \in \{16, 32, 64, 128, 256\}$ instances.