Systematic analysis of the impact of label noise correction on ML Fairness

TL;DR

This paper systematically evaluates how different label noise correction methods impact fairness in machine learning models trained on biased data, providing insights into their effectiveness and trade-offs.

Contribution

It introduces an empirical methodology to assess label noise correction techniques for fairness, applying it to six methods across multiple datasets.

Findings

Hybrid Label Noise Correction balances fairness and accuracy well

Clustering-Based Correction reduces discrimination most but lowers performance

Methodology can be applied to fairness benchmarks and standard datasets

Abstract

Arbitrary, inconsistent, or faulty decision-making raises serious concerns, and preventing unfair models is an increasingly important challenge in Machine Learning. Data often reflect past discriminatory behavior, and models trained on such data may reflect bias on sensitive attributes, such as gender, race, or age. One approach to developing fair models is to preprocess the training data to remove the underlying biases while preserving the relevant information, for example, by correcting biased labels. While multiple label noise correction methods are available, the information about their behavior in identifying discrimination is very limited. In this work, we develop an empirical methodology to systematically evaluate the effectiveness of label noise correction techniques in ensuring the fairness of models trained on biased datasets. Our methodology involves manipulating the amount of label noise and can be used with fairness benchmarks but also with standard ML datasets. We apply the methodology to analyze six label noise correction methods according to several fairness metrics on standard OpenML datasets. Our results suggest that the Hybrid Label Noise Correction method achieves the best trade-off between predictive performance and fairness. Clustering-Based Correction can reduce discrimination the most, however, at the cost of lower predictive performance.