BOOM: Benchmarking Out-Of-distribution Molecular Property Predictions of Machine Learning Models

TL;DR

BOOM introduces a comprehensive benchmark for evaluating the out-of-distribution molecular property prediction capabilities of machine learning models, revealing current limitations and guiding future improvements.

Contribution

This work provides the first systematic, chemically-informed benchmark for OOD molecular property prediction, evaluating over 150 models and analyzing factors affecting OOD generalization.

Findings

No model excels across all OOD tasks

Top models have 3x higher error OOD than in-distribution

Models with high inductive bias perform better on simple properties

Abstract

Data-driven molecular discovery leverages artificial intelligence/machine learning (AI/ML) and generative modeling to filter and design novel molecules. Discovering novel molecules requires accurate out-of-distribution (OOD) predictions, but ML models struggle to generalize OOD. Currently, no systematic benchmarks exist for molecular OOD prediction tasks. We present $\mathbf{BOOM}$, $\mathbf{b}$enchmarks for $\mathbf{o}$ut-$\mathbf{o}$f-distribution $\mathbf{m}$olecular property predictions: a chemically-informed benchmark for OOD performance on common molecular property prediction tasks. We evaluate over 150 model-task combinations to benchmark deep learning models on OOD performance. Overall, we find that no existing model achieves strong generalization across all tasks: even the top-performing model exhibited an average OOD error 3x higher than in-distribution. Current chemical foundation models do not show strong OOD extrapolation, while models with high inductive bias can perform well on OOD tasks with simple, specific properties. We perform extensive ablation experiments, highlighting how data generation, pre-training, hyperparameter optimization, model architecture, and molecular representation impact OOD performance. Developing models with strong OOD generalization is a new frontier challenge in chemical ML. This open-source benchmark is available at https://github.com/FLASK-LLNL/BOOM