Bayesian Quantification with Black-Box Estimators

TL;DR

This paper presents a Bayesian framework for class distribution estimation using black-box classifiers, offering a unified perspective, an efficient sampling scheme, and competitive results compared to existing methods.

Contribution

It introduces a Bayesian model that unifies various class distribution estimation algorithms and provides an efficient MCMC sampling method with asymptotic guarantees.

Findings

The Bayesian model closely relates to existing estimators.

The MCMC scheme is efficient and asymptotically consistent.

The model performs competitively or better than state-of-the-art methods.

Abstract

Understanding how different classes are distributed in an unlabeled data set is an important challenge for the calibration of probabilistic classifiers and uncertainty quantification. Approaches like adjusted classify and count, black-box shift estimators, and invariant ratio estimators use an auxiliary (and potentially biased) black-box classifier trained on a different (shifted) data set to estimate the class distribution and yield asymptotic guarantees under weak assumptions. We demonstrate that all these algorithms are closely related to the inference in a particular Bayesian model, approximating the assumed ground-truth generative process. Then, we discuss an efficient Markov Chain Monte Carlo sampling scheme for the introduced model and show an asymptotic consistency guarantee in the large-data limit. We compare the introduced model against the established point estimators in a variety of scenarios, and show it is competitive, and in some cases superior, with the state of the art.