Adaptive Sampling for Minimax Fair Classification

TL;DR

This paper introduces an adaptive sampling method to construct training datasets that improve fairness in classifiers, providing theoretical bounds and validating effectiveness on synthetic and real-world tasks.

Contribution

It proposes a novel adaptive sampling algorithm for fair classification, with theoretical performance bounds and practical heuristics for large-scale applications.

Findings

Adaptive sampling improves fairness in classifiers.

Theoretical bounds show optimality limits of the method.

Experimental results demonstrate effectiveness on diverse tasks.

Abstract

Machine learning models trained on uncurated datasets can often end up adversely affecting inputs belonging to underrepresented groups. To address this issue, we consider the problem of adaptively constructing training sets which allow us to learn classifiers that are fair in a minimax sense. We first propose an adaptive sampling algorithm based on the principle of optimism, and derive theoretical bounds on its performance. We also propose heuristic extensions of this algorithm suitable for application to large scale, practical problems. Next, by deriving algorithm independent lower-bounds for a specific class of problems, we show that the performance achieved by our adaptive scheme cannot be improved in general. We then validate the benefits of adaptively constructing training sets via experiments on synthetic tasks with logistic regression classifiers, as well as on several real-world tasks using convolutional neural networks (CNNs).