Can Active Learning Preemptively Mitigate Fairness Issues?

TL;DR

This paper investigates how active learning, especially uncertainty-based heuristics like BALD, can improve fairness in machine learning models by reducing dataset bias and enhancing predictive parity, with positive interactions observed when combined with fairness methods.

Contribution

The study demonstrates that uncertainty-based active learning improves fairness and accuracy, and explores beneficial interactions with fairness techniques like gradient reversal.

Findings

BALD improves predictive parity and accuracy over i.i.d. sampling.

Active learning interacts positively with fairness methods, enhancing overall fairness.

Combining BALD with fairness techniques yields better results on multiple benchmarks.

Abstract

Dataset bias is one of the prevailing causes of unfairness in machine learning. Addressing fairness at the data collection and dataset preparation stages therefore becomes an essential part of training fairer algorithms. In particular, active learning (AL) algorithms show promise for the task by drawing importance to the most informative training samples. However, the effect and interaction between existing AL algorithms and algorithmic fairness remain under-explored. In this paper, we study whether models trained with uncertainty-based AL heuristics such as BALD are fairer in their decisions with respect to a protected class than those trained with identically independently distributed (i.i.d.) sampling. We found a significant improvement on predictive parity when using BALD, while also improving accuracy compared to i.i.d. sampling. We also explore the interaction of algorithmic fairness methods such as gradient reversal (GRAD) and BALD. We found that, while addressing different fairness issues, their interaction further improves the results on most benchmarks and metrics we explored.