Prediction uncertainty validation for computational chemists

TL;DR

This paper introduces the calibration-sharpness framework for validating prediction uncertainty in computational chemistry, providing practical methods from simple checks to advanced statistics, adapted from meteorology.

Contribution

It adapts and demonstrates the calibration-sharpness framework for PU validation specifically in computational chemistry, including new concepts like tightness.

Findings

Methods validated on synthetic datasets

Applied to computational chemistry literature data

Provides a step-by-step validation approach

Abstract

Validation of prediction uncertainty (PU) is becoming an essential task for modern computational chemistry. Designed to quantify the reliability of predictions in meteorology, the calibration-sharpness (CS) framework is now widely used to optimize and validate uncertainty-aware machine learning (ML) methods. However, its application is not limited to ML and it can serve as a principled framework for any PU validation. The present article is intended as a step-by-step introduction to the concepts and techniques of PU validation in the CS framework, adapted to the specifics of computational chemistry. The presented methods range from elementary graphical checks to more sophisticated ones based on local calibration statistics. The concept of tightness, is introduced. The methods are illustrated on synthetic datasets and applied to uncertainty quantification data extracted from the computational chemistry literature.