Unsupervised Recalibration

TL;DR

Unsupervised recalibration (URC) enhances the accuracy of pre-trained probabilistic models on new, unlabeled data by detecting and correcting biases, especially across different subpopulations, without retraining or ground truth.

Contribution

This paper introduces URC, a novel method for improving model accuracy on new data without retraining or labeled data, effectively handling subpopulation biases.

Findings

URC detects when training data is unrepresentative of new data.

URC corrects biases without requiring ground truth.

URC reveals true subpopulation distributions.

Abstract

Unsupervised recalibration (URC) is a general way to improve the accuracy of an already trained probabilistic classification or regression model upon encountering new data while deployed in the field. URC does not require any ground truth associated with the new field data. URC merely observes the model's predictions and recognizes when the training set is not representative of field data, and then corrects to remove any introduced bias. URC can be particularly useful when applied separately to different subpopulations observed in the field that were not considered as features when training the machine learning model. This makes it possible to exploit subpopulation information without retraining the model or even having ground truth for some or all subpopulations available. Additionally, if these subpopulations are the object of study, URC serves to determine the correct ground truth distributions for them, where naive aggregation methods, like averaging the model's predictions, systematically underestimate their differences.