Calibration in Machine Learning Uncertainty Quantification: beyond consistency to target adaptivity

TL;DR

This paper emphasizes the importance of both consistency and adaptivity in uncertainty quantification for machine learning regression, proposing new validation methods to assess adaptivity beyond traditional calibration techniques.

Contribution

It introduces the concept of adaptivity as a key validation target in ML uncertainty quantification and proposes methods to evaluate it, complementing existing consistency assessments.

Findings

Consistency does not imply adaptivity in UQ.

Proposed validation methods effectively assess adaptivity.

Adaptivity is crucial for reliable predictions across feature space.

Abstract

Reliable uncertainty quantification (UQ) in machine learning (ML) regression tasks is becoming the focus of many studies in materials and chemical science. It is now well understood that average calibration is insufficient, and most studies implement additional methods testing the conditional calibration with respect to uncertainty, i.e. consistency. Consistency is assessed mostly by so-called reliability diagrams. There exists however another way beyond average calibration, which is conditional calibration with respect to input features, i.e. adaptivity. In practice, adaptivity is the main concern of the final users of a ML-UQ method, seeking for the reliability of predictions and uncertainties for any point in features space. This article aims to show that consistency and adaptivity are complementary validation targets, and that a good consistency does not imply a good adaptivity. Adapted validation methods are proposed and illustrated on a representative example.