When Can Memorization Improve Fairness?

TL;DR

This paper investigates how memorizing parts of a dataset can influence fairness metrics in multi-class classification, providing explicit bias expressions and conditions for bias elimination.

Contribution

It offers explicit formulas for bias due to memorization, characterizes datasets that eliminate bias, and bounds the data mass needed for fairness improvements.

Findings

Explicit bias expressions in terms of data distribution

Conditions for bias elimination through memorization

Bounds on dataset size for complete bias removal

Abstract

We study to which extent additive fairness metrics (statistical parity, equal opportunity and equalized odds) can be influenced in a multi-class classification problem by memorizing a subset of the population. We give explicit expressions for the bias resulting from memorization in terms of the label and group membership distribution of the memorized dataset and the classifier bias on the unmemorized dataset. We also characterize the memorized datasets that eliminate the bias for all three metrics considered. Finally we provide upper and lower bounds on the total probability mass in the memorized dataset that is necessary for the complete elimination of these biases.