Who's the (Multi-)Fairest of Them All: Rethinking Interpolation-Based Data Augmentation Through the Lens of Multicalibration

TL;DR

This paper critically examines interpolation-based data augmentation methods like Fair Mixup, revealing that vanilla Mixup often outperforms them in fairness and accuracy, especially when combined with multicalibration post-processing.

Contribution

It provides a rigorous evaluation of data augmentation methods using multicalibration, showing that simpler vanilla Mixup can outperform more complex fairness-oriented methods.

Findings

Vanilla Mixup outperforms Fair Mixup in fairness and accuracy.

Multicalibration post-processing can further improve fairness.

Fair Mixup often worsens performance compared to baseline.

Abstract

Data augmentation methods, especially SoTA interpolation-based methods such as Fair Mixup, have been widely shown to increase model fairness. However, this fairness is evaluated on metrics that do not capture model uncertainty and on datasets with only one, relatively large, minority group. As a remedy, multicalibration has been introduced to measure fairness while accommodating uncertainty and accounting for multiple minority groups. However, existing methods of improving multicalibration involve reducing initial training data to create a holdout set for post-processing, which is not ideal when minority training data is already sparse. This paper uses multicalibration to more rigorously examine data augmentation for classification fairness. We stress-test four versions of Fair Mixup on two structured data classification problems with up to 81 marginalized groups, evaluating multicalibration violations and balanced accuracy. We find that on nearly every experiment, Fair Mixup \textit{worsens} baseline performance and fairness, but the simple vanilla Mixup \textit{outperforms} both Fair Mixup and the baseline, especially when calibrating on small groups. \textit{Combining} vanilla Mixup with multicalibration post-processing, which enforces multicalibration through post-processing on a holdout set, further increases fairness.