Thermodynamic assessment of machine learning models for solid-state synthesis prediction

TL;DR

This paper evaluates machine learning models predicting solid-state material synthesis by comparing their predictions with thermodynamic calculations, revealing overpredictions and proposing a new assessment approach.

Contribution

It introduces a thermodynamic assessment framework for existing synthesis prediction models and highlights their limitations in predicting synthesizability.

Findings

Models tend to overpredict synthesizability.

Some scores correlate with thermodynamic stability.

Thermodynamic heuristics can inform model evaluation.

Abstract

Machine learning models have recently emerged to predict whether hypothetical solid-state materials can be synthesized. These models aim to circumvent direct first-principles modeling of solid-state phase transformations, instead learning from large databases of successfully synthesized materials. Here, we assess the alignment of several recently introduced synthesis prediction models with material and reaction thermodynamics, quantified by the energy with respect to the convex hull and a metric accounting for thermodynamic selectivity of enumerated synthesis reactions. A dataset of successful synthesis recipes was used to determine the likely bounds on both quantities beyond which materials can be deemed unlikely to be synthesized. With these bounds as context, thermodynamic quantities were computed using the CHGNet foundation potential for thousands of new hypothetical materials generated using the Chemeleon generative model. Four recently published machine learning models for synthesizability prediction were applied to this same dataset, and the resultant predictions were considered against computed thermodynamics. We find these models generally overpredict the likelihood of synthesis, but some model scores do trend with thermodynamic heuristics, assigning lower scores to materials that are less stable or do not have an available synthesis recipe that is calculated to be thermodynamically selective. In total, this work identifies existing gaps in machine learning models for materials synthesis and introduces a new approach to assess their quality in the absence of extensive negative examples (failed syntheses).