Binary Quantification and Dataset Shift: An Experimental Investigation

TL;DR

This paper investigates how current quantification algorithms perform under various types of dataset shift, revealing limitations and the need for more robust methods, through an extensive experimental analysis and a detailed taxonomy of dataset shifts.

Contribution

It introduces a fine-grained taxonomy of dataset shift types, establishes protocols for dataset generation under these shifts, and evaluates existing quantification methods across these scenarios.

Findings

Many methods robust to prior probability shift fail under other shifts

No current method is robust to all simulated dataset shifts

Quantification methods need development for broader applicability

Abstract

Quantification is the supervised learning task that consists of training predictors of the class prevalence values of sets of unlabelled data, and is of special interest when the labelled data on which the predictor has been trained and the unlabelled data are not IID, i.e., suffer from dataset shift. To date, quantification methods have mostly been tested only on a special case of dataset shift, i.e., prior probability shift; the relationship between quantification and other types of dataset shift remains, by and large, unexplored. In this work we carry out an experimental analysis of how current quantification algorithms behave under different types of dataset shift, in order to identify limitations of current approaches and hopefully pave the way for the development of more broadly applicable methods. We do this by proposing a fine-grained taxonomy of types of dataset shift, by establishing protocols for the generation of datasets affected by these types of shift, and by testing existing quantification methods on the datasets thus generated. One finding that results from this investigation is that many existing quantification methods that had been found robust to prior probability shift are not necessarily robust to other types of dataset shift. A second finding is that no existing quantification method seems to be robust enough to dealing with all the types of dataset shift we simulate in our experiments. The code needed to reproduce all our experiments is publicly available at https://github.com/pglez82/quant_datasetshift.