FedZMG: Efficient Client-Side Optimization in Federated Learning

TL;DR

FedZMG is a new client-side optimization method for federated learning that reduces client-drift and improves convergence speed and accuracy without extra communication or tuning, especially in non-IID data scenarios.

Contribution

The paper introduces FedZMG, a parameter-free, gradient regularization technique that enhances federated learning by addressing client-drift efficiently on resource-constrained devices.

Findings

FedZMG outperforms FedAvg and FedAdam in convergence speed.

FedZMG achieves higher final accuracy on non-IID datasets.

Theoretical analysis confirms reduced gradient variance and tighter convergence bounds.

Abstract

Federated Learning (FL) enables distributed model training on edge devices while preserving data privacy. However, clients tend to have non-Independent and Identically Distributed (non-IID) data, which often leads to client-drift, and therefore diminishing convergence speed and model performance. While adaptive optimizers have been proposed to mitigate these effects, they frequently introduce computational complexity or communication overhead unsuitable for resource-constrained IoT environments. This paper introduces Federated Zero Mean Gradients (FedZMG), a novel, parameter-free, client-side optimization algorithm designed to tackle client-drift by structurally regularizing the optimization space. Advancing the idea of Gradient Centralization, FedZMG projects local gradients onto a zero-mean hyperplane, effectively neutralizing the "intensity" or "bias" shifts inherent in heterogeneous data distributions without requiring additional communication or hyperparameter tuning. A theoretical analysis is provided, proving that FedZMG reduces the effective gradient variance and guarantees tighter convergence bounds compared to standard FedAvg. Extensive empirical evaluations on EMNIST, CIFAR100, and Shakespeare datasets demonstrate that FedZMG achieves better convergence speed and final validation accuracy compared to the baseline FedAvg and the adaptive optimizer FedAdam, particularly in highly non-IID settings.