A calibration test for evaluating set-based epistemic uncertainty representations

TL;DR

This paper introduces a novel statistical calibration test for set-based epistemic uncertainty representations, allowing instance-dependent convex combinations and leveraging proper scoring rules for improved calibration assessment.

Contribution

It proposes a new nonparametric calibration test that evaluates the validity of credal sets with instance-dependent combinations, enhancing uncertainty quantification methods.

Findings

The test effectively detects calibration issues in synthetic and real-world datasets.

Instance-dependent convex combinations improve calibration accuracy.

The method outperforms previous calibration assessment techniques.

Abstract

The accurate representation of epistemic uncertainty is a challenging yet essential task in machine learning. A widely used representation corresponds to convex sets of probabilistic predictors, also known as credal sets. One popular way of constructing these credal sets is via ensembling or specialized supervised learning methods, where the epistemic uncertainty can be quantified through measures such as the set size or the disagreement among members. In principle, these sets should contain the true data-generating distribution. As a necessary condition for this validity, we adopt the strongest notion of calibration as a proxy. Concretely, we propose a novel statistical test to determine whether there is a convex combination of the set's predictions that is calibrated in distribution. In contrast to previous methods, our framework allows the convex combination to be instance dependent, recognizing that different ensemble members may be better calibrated in different regions of the input space. Moreover, we learn this combination via proper scoring rules, which inherently optimize for calibration. Building on differentiable, kernel-based estimators of calibration errors, we introduce a nonparametric testing procedure and demonstrate the benefits of capturing instance-level variability on of synthetic and real-world experiments.