FLASH: Federated Learning Across Simultaneous Heterogeneities

TL;DR

FLASH introduces a novel client selection algorithm for federated learning that effectively manages multiple simultaneous heterogeneities, leading to significant accuracy improvements over existing methods.

Contribution

It is the first to unify handling data quality, distribution, and latency heterogeneities in federated learning using a contextual multi-armed bandit approach.

Findings

Achieves up to 10% absolute accuracy improvement.

Outperforms state-of-the-art FL frameworks under heterogeneity.

Enhances performance when combined with existing aggregation methods.

Abstract

The key premise of federated learning (FL) is to train ML models across a diverse set of data-owners (clients), without exchanging local data. An overarching challenge to this date is client heterogeneity, which may arise not only from variations in data distribution, but also in data quality, as well as compute/communication latency. An integrated view of these diverse and concurrent sources of heterogeneity is critical; for instance, low-latency clients may have poor data quality, and vice versa. In this work, we propose FLASH(Federated Learning Across Simultaneous Heterogeneities), a lightweight and flexible client selection algorithm that outperforms state-of-the-art FL frameworks under extensive sources of heterogeneity, by trading-off the statistical information associated with the client's data quality, data distribution, and latency. FLASH is the first method, to our knowledge, for handling all these heterogeneities in a unified manner. To do so, FLASH models the learning dynamics through contextual multi-armed bandits (CMAB) and dynamically selects the most promising clients. Through extensive experiments, we demonstrate that FLASH achieves substantial and consistent improvements over state-of-the-art baselines -- as much as 10% in absolute accuracy -- thanks to its unified approach. Importantly, FLASH also outperforms federated aggregation methods that are designed to handle highly heterogeneous settings and even enjoys a performance boost when integrated with them.