Fisher consistency for prior probability shift

TL;DR

This paper introduces Fisher consistency as an unbiasedness criterion for estimators of class prior probabilities, helping to identify reliable quantification methods under dataset shift.

Contribution

It formalizes Fisher consistency for class prior estimators and evaluates its implications for popular classifiers used in quantification tasks.

Findings

Adjusted Classify & Count is Fisher consistent.

EM-algorithm is Fisher consistent.

CDE-Iterate is not Fisher consistent.

Abstract

We introduce Fisher consistency in the sense of unbiasedness as a desirable property for estimators of class prior probabilities. Lack of Fisher consistency could be used as a criterion to dismiss estimators that are unlikely to deliver precise estimates in test datasets under prior probability and more general dataset shift. The usefulness of this unbiasedness concept is demonstrated with three examples of classifiers used for quantification: Adjusted Classify & Count, EM-algorithm and CDE-Iterate. We find that Adjusted Classify & Count and EM-algorithm are Fisher consistent. A counter-example shows that CDE-Iterate is not Fisher consistent and, therefore, cannot be trusted to deliver reliable estimates of class probabilities.