On the Quality of Uncertainty Estimates from Neural Network Potential Ensembles

TL;DR

This paper evaluates the effectiveness of neural network potential ensembles in estimating uncertainty, revealing they are often overconfident and need calibration, while Bayesian approaches can offer improved estimates.

Contribution

It systematically compares ensemble-based and Bayesian neural network potentials for uncertainty estimation across multiple case studies.

Findings

Ensembles often underestimate uncertainty and are overconfident.

Calibration improves the reliability of ensemble uncertainty estimates.

Bayesian NNPs provide more accurate uncertainty quantification.

Abstract

Neural network potentials (NNPs) combine the computational efficiency of classical interatomic potentials with the high accuracy and flexibility of the ab initio methods used to create the training set, but can also result in unphysical predictions when employed outside their training set distribution. Estimating the epistemic uncertainty of an NNP is required in active learning or on-the-fly generation of potentials. Inspired from their use in other machine-learning applications, NNP ensembles have been used for uncertainty prediction in several studies, with the caveat that ensembles do not provide a rigorous Bayesian estimate of the uncertainty. To test whether NNP ensembles provide accurate uncertainty estimates, we train such ensembles in four different case studies, and compare the predicted uncertainty with the errors on out-of-distribution validation sets. Our results indicate that NNP ensembles are often overconfident, underestimating the uncertainty of the model, and require to be calibrated for each system and architecture. We also provide evidence that Bayesian NNPs, obtained by sampling the posterior distribution of the model parameters using Monte-Carlo techniques, can provide better uncertainty estimates.