Robust model benchmarking and bias-imbalance in data-driven materials science: a case study on MODNet

TL;DR

This paper benchmarks the MODNet approach against MatBench datasets, highlighting its strengths, limitations, and the importance of evaluating model uncertainty and bias for reliable materials science predictions.

Contribution

It introduces a comprehensive benchmarking of MODNet on MatBench, emphasizing the need for diverse metrics and uncertainty quantification in model evaluation.

Findings

MODNet outperforms on 6 of 13 tasks

MODNet performs well with less than 10,000 samples

Uncertainty assessment reveals impact of data bias and imbalance

Abstract

As the number of novel data-driven approaches to material science continues to grow, it is crucial to perform consistent quality, reliability and applicability assessments of model performance. In this paper, we benchmark the Materials Optimal Descriptor Network (MODNet) method and architecture against the recently released MatBench v0.1, a curated test suite of materials datasets. MODNet is shown to outperform current leaders on 6 of the 13 tasks, whilst closely matching the current leaders on a further 2 tasks; MODNet performs particularly well when the number of samples is below 10,000. Attention is paid to two topics of concern when benchmarking models. First, we encourage the reporting of a more diverse set of metrics as it leads to a more comprehensive and holistic comparison of model performance. Second, an equally important task is the uncertainty assessment of a model towards a target domain. Significant variations in validation errors can be observed, depending on the imbalance and bias in the training set (i.e., similarity between training and application space). By using an ensemble MODNet model, confidence intervals can be built and the uncertainty on individual predictions can be quantified. Imbalance and bias issues are often overlooked, and yet are important for successful real-world applications of machine learning in materials science and condensed matter.