FeDa4Fair: Client-Level Federated Datasets for Fairness Evaluation

TL;DR

FeDa4Fair is a benchmarking framework that creates datasets to evaluate fairness in federated learning under heterogeneous client biases, addressing limitations of existing fairness solutions.

Contribution

It introduces a library for generating tailored datasets, a benchmark suite for evaluation, and functions for assessing fairness outcomes in federated learning.

Findings

First benchmark for fairness in heterogeneous FL scenarios.

Supports evaluation of fairness methods under conflicting client biases.

Provides standardized datasets and evaluation tools.

Abstract

Federated Learning (FL) enables collaborative training while preserving privacy, yet it introduces a critical challenge: the "illusion of fairness''. A global model, usually evaluated on the server, appears fair on average while keeping persistent discrimination at the client level. Current fairness-enhancing FL solutions often fall short, as they typically mitigate biases for a single, usually binary, sensitive attribute, while ignoring two realistic and conflicting scenarios: attribute-bias (where clients are unfair toward different sensitive attributes) and value-bias (where clients exhibit conflicting biases toward different values of the same attribute). To support more robust and reproducible fairness research in FL, we introduce FeDa4Fair, the first benchmarking framework designed to stress-test fairness methods under these heterogeneous conditions. Our contributions are three-fold: (1) We introduce FeDa4Fair, a library designed to create datasets tailored to evaluating fair FL methods under heterogeneous client bias; (2) we release a benchmark suite generated by the FeDa4Fair library to standardize the evaluation of fair FL methods; (3) we provide ready-to-use functions for evaluating fairness outcomes for these datasets.