Tackling the Non-IID Issue in Heterogeneous Federated Learning by Gradient Harmonization

TL;DR

This paper introduces FedGH, a gradient harmonization method that reduces gradient conflicts caused by data heterogeneity in federated learning, leading to improved model performance across various non-IID scenarios.

Contribution

The paper proposes FedGH, a novel gradient projection technique that mitigates local drift and conflicts in federated learning with non-IID data, enhancing existing methods without extra tuning.

Findings

FedGH improves performance of state-of-the-art FL models.

The method is especially effective under strong heterogeneity.

FedGH is easy to integrate into existing FL frameworks.

Abstract

Federated learning (FL) is a privacy-preserving paradigm for collaboratively training a global model from decentralized clients. However, the performance of FL is hindered by non-independent and identically distributed (non-IID) data and device heterogeneity. In this work, we revisit this key challenge through the lens of gradient conflicts on the server side. Specifically, we first investigate the gradient conflict phenomenon among multiple clients and reveal that stronger heterogeneity leads to more severe gradient conflicts. To tackle this issue, we propose FedGH, a simple yet effective method that mitigates local drifts through Gradient Harmonization. This technique projects one gradient vector onto the orthogonal plane of the other within conflicting client pairs. Extensive experiments demonstrate that FedGH consistently enhances multiple state-of-the-art FL baselines across diverse benchmarks and non-IID scenarios. Notably, FedGH yields more significant improvements in scenarios with stronger heterogeneity. As a plug-and-play module, FedGH can be seamlessly integrated into any FL framework without requiring hyperparameter tuning.