Demystifying the Effects of Non-Independence in Federated Learning

TL;DR

This paper investigates how non-independent data sampling methods, like block-cyclic sampling, impact federated learning's accuracy, fairness, and convergence, revealing robustness to some patterns but significant degradation with others.

Contribution

It provides an empirical analysis of the effects of block-cyclic sampling and unbalanced data distributions on federated learning performance.

Findings

Robustness to two-block cyclic sampling over time zones.

Performance drops up to 26% with multi-block dependent sampling.

Unbalanced block distributions further impair convergence and accuracy.

Abstract

Federated Learning (FL) enables statistical models to be built on user-generated data without compromising data security and user privacy. For this reason, FL is well suited for on-device learning from mobile devices where data is abundant and highly privatized. Constrained by the temporal availability of mobile devices, only a subset of devices is accessible to participate in the iterative protocol consisting of training and aggregation. In this study, we take a step toward better understanding the effect of non-independent data distributions arising from block-cyclic sampling. By conducting extensive experiments on visual classification, we measure the effects of block-cyclic sampling (both standalone and in combination with non-balanced block distributions). Specifically, we measure the alterations induced by block-cyclic sampling from the perspective of accuracy, fairness, and convergence rate. Experimental results indicate robustness to cycling over a two-block structure, e.g., due to time zones. In contrast, drawing data samples dependently from a multi-block structure significantly degrades the performance and rate of convergence by up to 26%. Moreover, we find that this performance degeneration is further aggravated by unbalanced block distributions to a point that can no longer be adequately compensated by higher communication and more frequent synchronization.