Federated Learning on Non-IID Data Silos: An Experimental Study

TL;DR

This paper systematically evaluates federated learning algorithms on various non-IID data partitioning strategies, revealing significant challenges and the lack of a universally best algorithm, thereby aiding future research in data silo scenarios.

Contribution

It introduces comprehensive non-IID data partitioning strategies and provides an extensive experimental comparison of FL algorithms under these conditions.

Findings

Non-IID data significantly impacts FL accuracy.

No single FL algorithm outperforms others across all non-IID scenarios.

Insights provided for future research on data silos.

Abstract

Due to the increasing privacy concerns and data regulations, training data have been increasingly fragmented, forming distributed databases of multiple "data silos" (e.g., within different organizations and countries). To develop effective machine learning services, there is a must to exploit data from such distributed databases without exchanging the raw data. Recently, federated learning (FL) has been a solution with growing interests, which enables multiple parties to collaboratively train a machine learning model without exchanging their local data. A key and common challenge on distributed databases is the heterogeneity of the data distribution among the parties. The data of different parties are usually non-independently and identically distributed (i.e., non-IID). There have been many FL algorithms to address the learning effectiveness under non-IID data settings. However, there lacks an experimental study on systematically understanding their advantages and disadvantages, as previous studies have very rigid data partitioning strategies among parties, which are hardly representative and thorough. In this paper, to help researchers better understand and study the non-IID data setting in federated learning, we propose comprehensive data partitioning strategies to cover the typical non-IID data cases. Moreover, we conduct extensive experiments to evaluate state-of-the-art FL algorithms. We find that non-IID does bring significant challenges in learning accuracy of FL algorithms, and none of the existing state-of-the-art FL algorithms outperforms others in all cases. Our experiments provide insights for future studies of addressing the challenges in "data silos".