On the Power of Adaptive Weighted Aggregation in Heterogeneous Federated Learning and Beyond

TL;DR

This paper investigates the limitations of FedAvg in heterogeneous federated learning, introduces a new aggregation strategy called FedAWARE, and demonstrates its superior convergence and generalization performance through theoretical analysis and extensive experiments.

Contribution

It provides a new theoretical measure of client consensus, proposes FedAWARE as an improved aggregation method, and validates its effectiveness in heterogeneous FL scenarios.

Findings

FedAWARE achieves faster convergence in heterogeneous settings.

FedAWARE improves generalization performance of FL algorithms.

Theoretical analysis explains the success of FedAWARE under heterogeneity.

Abstract

Federated averaging (FedAvg) is the most fundamental algorithm in Federated learning (FL). Previous theoretical results assert that FedAvg convergence and generalization degenerate under heterogeneous clients. However, recent empirical results show that FedAvg can perform well in many real-world heterogeneous tasks. These results reveal an inconsistency between FL theory and practice that is not fully explained. In this paper, we show that common heterogeneity measures contribute to this inconsistency based on rigorous convergence analysis. Furthermore, we introduce a new measure \textit{client consensus dynamics} and prove that \textit{FedAvg can effectively handle client heterogeneity when an appropriate aggregation strategy is used}. Building on this theoretical insight, we present a simple and effective FedAvg variant termed FedAWARE. Extensive experiments on three datasets and two modern neural network architectures demonstrate that FedAWARE ensures faster convergence and better generalization in heterogeneous client settings. Moreover, our results show that FedAWARE can significantly enhance the generalization performance of advanced FL algorithms when used as a plug-in module.