Federated learning with incremental clustering for heterogeneous data

TL;DR

This paper introduces FLIC, a federated learning method that incrementally clusters clients based on their updates, improving performance on heterogeneous data without extra communication, and demonstrating robustness against poisoning attacks.

Contribution

FLIC enables client clustering during federated training using update information, avoiding the need for simultaneous parameter transmission and enhancing robustness against malicious clients.

Findings

Successfully clusters clients with similar data distributions in non-IID scenarios.

Maintains performance without additional communication overhead.

Provides robustness against poisoning attacks even with over 50% malicious clients.

Abstract

Federated learning enables different parties to collaboratively build a global model under the orchestration of a server while keeping the training data on clients' devices. However, performance is affected when clients have heterogeneous data. To cope with this problem, we assume that despite data heterogeneity, there are groups of clients who have similar data distributions that can be clustered. In previous approaches, in order to cluster clients the server requires clients to send their parameters simultaneously. However, this can be problematic in a context where there is a significant number of participants that may have limited availability. To prevent such a bottleneck, we propose FLIC (Federated Learning with Incremental Clustering), in which the server exploits the updates sent by clients during federated training instead of asking them to send their parameters simultaneously. Hence no additional communications between the server and the clients are necessary other than what classical federated learning requires. We empirically demonstrate for various non-IID cases that our approach successfully splits clients into groups following the same data distributions. We also identify the limitations of FLIC by studying its capability to partition clients at the early stages of the federated learning process efficiently. We further address attacks on models as a form of data heterogeneity and empirically show that FLIC is a robust defense against poisoning attacks even when the proportion of malicious clients is higher than 50\%.