Materials Property Prediction with Uncertainty Quantification: A Benchmark Study

TL;DR

This study evaluates various uncertainty quantification methods for graph neural networks in materials property prediction, revealing that ensemble methods are not always optimal, and provides open-source tools for the community.

Contribution

It offers a comprehensive benchmark of UQ methods in materials property prediction, highlighting their effectiveness and limitations, and supplies accessible code and datasets.

Findings

Ensemble methods are not the best for UQ in this context.

Evaluation over four crystal datasets demonstrates varied UQ method performance.

Open-source code and datasets are provided for community use.

Abstract

Uncertainty quantification (UQ) has increasing importance in building robust high-performance and generalizable materials property prediction models. It can also be used in active learning to train better models by focusing on getting new training data from uncertain regions. There are several categories of UQ methods each considering different types of uncertainty sources. Here we conduct a comprehensive evaluation on the UQ methods for graph neural network based materials property prediction and evaluate how they truly reflect the uncertainty that we want in error bound estimation or active learning. Our experimental results over four crystal materials datasets (including formation energy, adsorption energy, total energy, and band gap properties) show that the popular ensemble methods for uncertainty estimation is NOT the best choice for UQ in materials property prediction. For the convenience of the community, all the source code and data sets can be accessed freely at \url{https://github.com/usccolumbia/materialsUQ}.