Re-Assessing the "Classify and Count" Quantification Method

TL;DR

This paper reevaluates the Classify and Count method for quantification, demonstrating that with proper hyperparameter tuning and true quantification loss, CC can achieve near-state-of-the-art accuracy, challenging prior assumptions.

Contribution

It shows that properly optimized CC methods can perform competitively with advanced quantification techniques, emphasizing the importance of hyperparameter tuning and appropriate loss functions.

Findings

Optimized CC achieves near-state-of-the-art accuracy.

Hyperparameter tuning with quantification loss improves CC performance.

Experiments on sentiment datasets support the revised assessment.

Abstract

Learning to quantify (a.k.a.\ quantification) is a task concerned with training unbiased estimators of class prevalence via supervised learning. This task originated with the observation that "Classify and Count" (CC), the trivial method of obtaining class prevalence estimates, is often a biased estimator, and thus delivers suboptimal quantification accuracy; following this observation, several methods for learning to quantify have been proposed that have been shown to outperform CC. In this work we contend that previous works have failed to use properly optimised versions of CC. We thus reassess the real merits of CC (and its variants), and argue that, while still inferior to some cutting-edge methods, they deliver near-state-of-the-art accuracy once (a) hyperparameter optimisation is performed, and (b) this optimisation is performed by using a true quantification loss instead of a standard classification-based loss. Experiments on three publicly available binary sentiment classification datasets support these conclusions.