Holistic chemical evaluation reveals pitfalls in reaction prediction models

TL;DR

This paper introduces a comprehensive evaluation framework for chemical reaction prediction models, highlighting their limitations in stereoselectivity and out-of-distribution generalization, to improve robustness and chemical discovery.

Contribution

It presents CHORISO, a curated dataset with tailored splits and new holistic metrics, enabling more detailed assessment of reaction prediction models beyond simple accuracy.

Findings

State-of-the-art models show limitations in stereoselectivity.

Models struggle with chemical out-of-distribution generalization.

Holistic evaluation reveals critical model weaknesses.

Abstract

The prediction of chemical reactions has gained significant interest within the machine learning community in recent years, owing to its complexity and crucial applications in chemistry. However, model evaluation for this task has been mostly limited to simple metrics like top-k accuracy, which obfuscates fine details of a model's limitations. Inspired by progress in other fields, we propose a new assessment scheme that builds on top of current approaches, steering towards a more holistic evaluation. We introduce the following key components for this goal: CHORISO, a curated dataset along with multiple tailored splits to recreate chemically relevant scenarios, and a collection of metrics that provide a holistic view of a model's advantages and limitations. Application of this method to state-of-the-art models reveals important differences on sensitive fronts, especially stereoselectivity and chemical out-of-distribution generalization. Our work paves the way towards robust prediction models that can ultimately accelerate chemical discovery.