FLAGS Framework for Comparative Analysis of Federated Learning Algorithms

TL;DR

This paper presents FLAGS, a comprehensive framework for evaluating various federated learning algorithms under diverse conditions, revealing insights into their performance and robustness, especially in decentralized and heterogeneous environments.

Contribution

The work introduces a unified simulation framework for fair comparison of multiple FL algorithms and provides an extensive analysis of their performance across different scenarios.

Findings

Decentralized FL algorithms achieve comparable accuracy to centralized ones.

Decentralized FL performs well in noisy environments and with high local update rates.

Extreme data skewness adversely affects decentralized FL more than centralized variants.

Abstract

Federated Learning (FL) has become a key choice for distributed machine learning. Initially focused on centralized aggregation, recent works in FL have emphasized greater decentralization to adapt to the highly heterogeneous network edge. Among these, Hierarchical, Device-to-Device and Gossip Federated Learning (HFL, D2DFL \& GFL respectively) can be considered as foundational FL algorithms employing fundamental aggregation strategies. A number of FL algorithms were subsequently proposed employing multiple fundamental aggregation schemes jointly. Existing research, however, subjects the FL algorithms to varied conditions and gauges the performance of these algorithms mainly against Federated Averaging (FedAvg) only. This work consolidates the FL landscape and offers an objective analysis of the major FL algorithms through a comprehensive cross-evaluation for a wide range of operating conditions. In addition to the three foundational FL algorithms, this work also analyzes six derived algorithms. To enable a uniform assessment, a multi-FL framework named FLAGS: Federated Learning AlGorithms Simulation has been developed for rapid configuration of multiple FL algorithms. Our experiments indicate that fully decentralized FL algorithms achieve comparable accuracy under multiple operating conditions, including asynchronous aggregation and the presence of stragglers. Furthermore, decentralized FL can also operate in noisy environments and with a comparably higher local update rate. However, the impact of extremely skewed data distributions on decentralized FL is much more adverse than on centralized variants. The results indicate that it may not be necessary to restrict the devices to a single FL algorithm; rather, multi-FL nodes may operate with greater efficiency.