On the Interconnections of Calibration, Quantification, and Classifier Accuracy Prediction under Dataset Shift

TL;DR

This paper explores the deep connections between calibration, quantification, and classifier accuracy prediction under dataset shift, demonstrating their equivalence and proposing unified methods that outperform specialized approaches.

Contribution

It proves the equivalence of calibration, quantification, and accuracy prediction under dataset shift and introduces cross-disciplinary methods that enhance performance.

Findings

Unified methods often outperform dedicated approaches

Proven equivalence enables cross-disciplinary solutions

Encourages development of integrated approaches

Abstract

When the distribution of the data used to train a classifier differs from that of the test data, i.e., under dataset shift, well-established routines for calibrating the decision scores of the classifier, estimating the proportion of positives in a test sample, or estimating the accuracy of the classifier, become particularly challenging. This paper investigates the interconnections among three fundamental problems, calibration, quantification, and classifier accuracy prediction, under dataset shift conditions. Specifically, we prove their equivalence through mutual reduction, i.e., we show that access to an oracle for any one of these tasks enables the resolution of the other two. Based on these proofs, we propose new methods for each problem based on direct adaptations of well-established methods borrowed from the other disciplines. Our results show such methods are often competitive, and sometimes even surpass the performance of dedicated approaches from each discipline. The main goal of this paper is to fostering cross-fertilization among these research areas, encouraging the development of unified approaches and promoting synergies across the fields.