Deep Learning Foundation Models from Classical Molecular Descriptors

TL;DR

This paper introduces CheMeleon, a large foundation model based on classical molecular descriptors, which outperforms traditional machine learning methods on molecular property prediction benchmarks.

Contribution

CheMeleon is the first foundation model to surpass classical methods using low-noise molecular descriptors for pre-training, demonstrating significant performance improvements.

Findings

Achieves 75% win rate on Polaris benchmarks

Achieves 97% win rate on MoleculeACE assays

Outperforms Random Forest, fastprop, and Chemprop models

Abstract

Fast and accurate data-driven prediction of molecular properties is pivotal to scientific advancements across myriad chemical domains. Deep learning methods have recently garnered much attention, despite their inability to outperform classical machine learning methods when tested on practical, real-world benchmarks with limited training data. This study seeks to bridge this gap with CheMeleon, a O(10M) parameter foundation model that enables directed message-passing neural networks to finally exceed the performance of classical methods. Evaluated on 58 benchmark datasets from Polaris and MoleculeACE, CheMeleon achieves a win rate of 75% on Polaris tasks, outperforming baselines like Random Forest (68%), fastprop (36%), and Chemprop (32%), and a 97% win rate on MoleculeACE assays, surpassing Random Forest (50%) and other foundation models. Unlike conventional pre-training approaches that rely on noisy experimental data or biased quantum mechanical simulations, CheMeleon utilizes low-noise molecular descriptors to learn rich and highly transferable molecular representations, suggesting a new avenue for foundation model pre-training.