The Benefits and Risks of Transductive Approaches for AI Fairness

TL;DR

This paper investigates how the composition of holdout sets in transductive learning impacts AI fairness, revealing that balanced and representative holdouts can improve fairness metrics and mitigate disparities.

Contribution

It highlights the importance of holdout set composition in transductive learning for fairness, an aspect largely overlooked in prior research.

Findings

Imbalanced holdout sets worsen fairness disparities.

Balanced holdout sets can reduce biases from training data.

Holdout set diversity is crucial for fair AI models.

Abstract

Recently, transductive learning methods, which leverage holdout sets during training, have gained popularity for their potential to improve speed, accuracy, and fairness in machine learning models. Despite this, the composition of the holdout set itself, particularly the balance of sensitive sub-groups, has been largely overlooked. Our experiments on CIFAR and CelebA datasets show that compositional changes in the holdout set can substantially influence fairness metrics. Imbalanced holdout sets exacerbate existing disparities, while balanced holdouts can mitigate issues introduced by imbalanced training data. These findings underline the necessity of constructing holdout sets that are both diverse and representative.