FedSkip: Combatting Statistical Heterogeneity with Federated Skip Aggregation

TL;DR

FedSkip is a novel federated learning method that improves client model optima in heterogeneous data environments by periodically skipping aggregation, leading to higher accuracy and better efficiency.

Contribution

This paper introduces FedSkip, a data-driven approach that enhances federated learning performance by addressing client optima issues in non-IID settings through periodic skipping.

Findings

FedSkip achieves higher accuracy than existing methods.

It improves aggregation and communication efficiency.

Experimental results validate its effectiveness across datasets.

Abstract

The statistical heterogeneity of the non-independent and identically distributed (non-IID) data in local clients significantly limits the performance of federated learning. Previous attempts like FedProx, SCAFFOLD, MOON, FedNova and FedDyn resort to an optimization perspective, which requires an auxiliary term or re-weights local updates to calibrate the learning bias or the objective inconsistency. However, in addition to previous explorations for improvement in federated averaging, our analysis shows that another critical bottleneck is the poorer optima of client models in more heterogeneous conditions. We thus introduce a data-driven approach called FedSkip to improve the client optima by periodically skipping federated averaging and scattering local models to the cross devices. We provide theoretical analysis of the possible benefit from FedSkip and conduct extensive experiments on a range of datasets to demonstrate that FedSkip achieves much higher accuracy, better aggregation efficiency and competing communication efficiency. Source code is available at: https://github.com/MediaBrain-SJTU/FedSkip.