Improving Federated Learning Communication Efficiency with Global Momentum Fusion for Gradient Compression Schemes

TL;DR

This paper introduces Global Momentum Fusion, a novel gradient compression scheme for federated learning that reduces communication costs and maintains accuracy despite non-IID data distributions.

Contribution

The paper proposes a new compression compensation method called Global Momentum Fusion that improves communication efficiency and robustness in federated learning with non-IID data.

Findings

Reduces communication overheads in federated learning.

Maintains comparable model accuracy with non-IID data.

Effective in a hub-and-spoke network topology.

Abstract

Communication costs within Federated learning hinder the system scalability for reaching more data from more clients. The proposed FL adopts a hub-and-spoke network topology. All clients communicate through the central server. Hence, reducing communication overheads via techniques such as data compression has been proposed to mitigate this issue. Another challenge of federated learning is unbalanced data distribution, data on each client are not independent and identically distributed (non-IID) in a typical federated learning setting. In this paper, we proposed a new compression compensation scheme called Global Momentum Fusion (GMF) which reduces communication overheads between FL clients and the server and maintains comparable model accuracy in the presence of non-IID data. GitHub repository: https://github.com/tony92151/global-momentum-fusion-fl