Estimating calibration error under label shift without labels

TL;DR

This paper introduces a new method for estimating calibration error in machine learning models under label shift without requiring target domain labels, ensuring reliable calibration assessment in deployment scenarios.

Contribution

It proposes a novel importance re-weighting based estimator for calibration error under label shift, which is consistent and asymptotically unbiased without target labels.

Findings

Effective across diverse real-world datasets

Reliable under various label-shift conditions

Outperforms existing calibration estimators

Abstract

In the face of dataset shift, model calibration plays a pivotal role in ensuring the reliability of machine learning systems. Calibration error (CE) is an indicator of the alignment between the predicted probabilities and the classifier accuracy. While prior works have delved into the implications of dataset shift on calibration, existing CE estimators assume access to labels from the target domain, which are often unavailable in practice, i.e., when the model is deployed and used. This work addresses such challenging scenario, and proposes a novel CE estimator under label shift, which is characterized by changes in the marginal label distribution $p(Y)$, while keeping the conditional $p(X|Y)$ constant between the source and target distributions. Our contribution is an approach, which, by leveraging importance re-weighting of the labeled source distribution, provides consistent and asymptotically unbiased CE estimation with respect to the shifted target distribution. Empirical results across diverse real-world datasets, under various conditions and label-shift intensities, demonstrate the effectiveness and reliability of the proposed estimator.