FedGPS: Statistical Rectification Against Data Heterogeneity in Federated Learning

TL;DR

FedGPS introduces a robust federated learning framework that leverages statistical and gradient information to effectively address data heterogeneity, improving model performance across diverse scenarios.

Contribution

The paper proposes FedGPS, a novel method that integrates statistical distribution and gradient information to enhance robustness against data heterogeneity in federated learning.

Findings

FedGPS outperforms existing methods in various heterogeneity scenarios.

Sharing statistical information mitigates data heterogeneity effects.

FedGPS demonstrates improved robustness and convergence.

Abstract

Federated Learning (FL) confronts a significant challenge known as data heterogeneity, which impairs model performance and convergence. Existing methods have made notable progress in addressing this issue. However, improving performance in certain heterogeneity scenarios remains an overlooked question: \textit{How robust are these methods to deploy under diverse heterogeneity scenarios?} To answer this, we conduct comprehensive evaluations across varied heterogeneity scenarios, showing that most existing methods exhibit limited robustness. Meanwhile, insights from these experiments highlight that sharing statistical information can mitigate heterogeneity by enabling clients to update with a global perspective. Motivated by this, we propose \textbf{FedGPS} (\textbf{Fed}erated \textbf{G}oal-\textbf{P}ath \textbf{S}ynergy), a novel framework that seamlessly integrates statistical distribution and gradient information from others. Specifically, FedGPS statically modifies each client's learning objective to implicitly model the global data distribution using surrogate information, while dynamically adjusting local update directions with gradient information from other clients at each round. Extensive experiments show that FedGPS outperforms state-of-the-art methods across diverse heterogeneity scenarios, validating its effectiveness and robustness. The code is available at: https://github.com/CUHK-AIM-Group/FedGPS.