An Experimental Study of Different Aggregation Schemes in Semi-Asynchronous Federated Learning

TL;DR

This paper compares two aggregation schemes in semi-asynchronous federated learning, revealing significant differences in accuracy, convergence speed, and stability across various tasks.

Contribution

It systematically evaluates the performance gap between FedSGD and FedAvg in SAFL, providing insights into their respective advantages and limitations.

Findings

FedSGD achieves higher accuracy and faster convergence.

FedAvg handles stragglers better but converges slower.

Performance gaps vary across different task scenarios.

Abstract

Federated learning is highly valued due to its high-performance computing in distributed environments while safeguarding data privacy. To address resource heterogeneity, researchers have proposed a semi-asynchronous federated learning (SAFL) architecture. However, the performance gap between different aggregation targets in SAFL remain unexplored. In this paper, we systematically compare the performance between two algorithm modes, FedSGD and FedAvg that correspond to aggregating gradients and models, respectively. Our results across various task scenarios indicate these two modes exhibit a substantial performance gap. Specifically, FedSGD achieves higher accuracy and faster convergence but experiences more severe fluctuates in accuracy, whereas FedAvg excels in handling straggler issues but converges slower with reduced accuracy.