A Bayesian Approach for Accurate Classification-Based Aggregates

TL;DR

This paper introduces a Bayesian bias correction method for classification-based aggregate estimates, effectively addressing estimation inaccuracies and negative value issues, especially in small sample scenarios with class imbalance.

Contribution

The paper presents a novel Bayesian inference approach that incorporates parameter constraints to improve bias correction in classification-based aggregate estimation.

Findings

Outperforms existing methods in mean squared error on real-world data

Effectively handles class imbalance and small sample size issues

Addresses estimation inaccuracies and negative estimate problems

Abstract

In this paper, we study the accuracy of values aggregated over classes predicted by a classification algorithm. The problem is that the resulting aggregates (e.g., sums of a variable) are known to be biased. The bias can be large even for highly accurate classification algorithms, in particular when dealing with class-imbalanced data. To correct this bias, the algorithm's classification error rates have to be estimated. In this estimation, two issues arise when applying existing bias correction methods. First, inaccuracies in estimating classification error rates have to be taken into account. Second, impermissible estimates, such as a negative estimate for a positive value, have to be dismissed. We show that both issues are relevant in applications where the true labels are known only for a small set of data points. We propose a novel bias correction method using Bayesian inference. The novelty of our method is that it imposes constraints on the model parameters. We show that our method solves the problem of biased classification-based aggregates as well as the two issues above, in the general setting of multi-class classification. In the empirical evaluation, using a binary classifier on a real-world dataset of company tax returns, we show that our method outperforms existing methods in terms of mean squared error.