Methods for comparing uncertainty quantifications for material property predictions

TL;DR

This paper introduces a set of metrics and figures from machine learning to evaluate the quality of uncertainty estimates in material property predictions, demonstrated through a case study on adsorption energies.

Contribution

It provides a standardized suite of tools for assessing uncertainty estimates in computational materials science, addressing a current lack of evaluation procedures.

Findings

The proposed metrics effectively evaluate uncertainty quality.

A CNN-Gaussian process model outperforms other methods.

The suite aids in selecting reliable predictive models.

Abstract

Data science and informatics tools have been proliferating recently within the computational materials science and catalysis fields. This proliferation has spurned the creation of various frameworks for automated materials screening, discovery, and design. Underpinning these frameworks are surrogate models with uncertainty estimates on their predictions. These uncertainty estimates are instrumental for determining which materials to screen next, but the computational catalysis field does not yet have a standard procedure for judging the quality of such uncertainty estimates. Here we present a suite of figures and performance metrics derived from the machine learning community that can be used to judge the quality of such uncertainty estimates. This suite probes the accuracy, calibration, and sharpness of a model quantitatively. We then show a case study where we judge various methods for predicting density-functional-theory-calculated adsorption energies. Of the methods studied here, we find that the best performer is a model where a convolutional neural network is used to supply features to a Gaussian process regressor, which then makes predictions of adsorption energies along with corresponding uncertainty estimates.