A robust sampling technique for realistic distribution simulation in federated learning

TL;DR

This paper introduces a sampling method to simulate realistic data distributions in federated learning, showing how client biases can degrade model performance.

Contribution

A novel sampling algorithm using chi-squared and Gini impurity to simulate non-IID label distributions in federated learning.

Findings

The proposed sampling technique realistically simulates client-biased label distributions.

A global model's performance dropped by 25.3% on weight and 28.7% on height estimation due to biased distributions.

The method is architecture-agnostic and applicable to various non-IID problems.

Abstract

Federated Learning helps training deep learning networks with diverse data from different locations, particularly in restricted clinical settings. However, label distributions overlapping only partially across clients, due to different demographics, may significantly harm the global training, and thus local model performance. Investigating such effects before rolling out large-scale Federated Learning setups requires proper sampling of the expected label distributions.

We present a sampling algorithm to build data subsets according to desired mean and standard deviations from an initial global distribution. To this end, we incorporate the chi-squared and Gini impurity measures to numerically optimize label distributions for multiple groups in an efficient fashion.

Using a real-world application scenario, we sample train and test groups according to region-specific distributions for 3D camera-based weight and height estimation in a clinical context, comparing a hard data split serving as a baseline with our proposed sampling technique. We train a baseline model on all data for comparison and use Federated Averaging to combine the training of our data subsets, demonstrating a realistic deterioration of 25.3 % on weight and 28.7 % on height estimations by the global model.

Realistically client-biased label distribution can notably harm the training in a federated context. Our sampling algorithm for simulating realistic data distributions opens up an efficient way for prior analysis of this effect. The technique is agnostic to the chosen network architecture and target scenario and can be adapted to any feature or label problem with non-IID subpopulations.

Figures