Depersonalized Federated Learning: Tackling Statistical Heterogeneity by Alternating Stochastic Gradient Descent

TL;DR

This paper introduces a novel federated learning method that uses alternating stochastic gradient descent and depersonalization to address statistical heterogeneity, improving convergence speed and stability in non-iid data environments.

Contribution

The paper proposes a new federated learning approach that decouples global and local objectives with alternating SGD and depersonalization, enhancing convergence in heterogeneous data settings.

Findings

Accelerates convergence in non-iid data scenarios.

Reduces variance in local updates.

Proven to converge at a sublinear rate in non-convex settings.

Abstract

Federated learning (FL), which has gained increasing attention recently, enables distributed devices to train a common machine learning (ML) model for intelligent inference cooperatively without data sharing. However, problems in practical networks, such as non-independent-and-identically-distributed (non-iid) raw data and limited bandwidth, give rise to slow and unstable convergence of the FL training process. To address these issues, we propose a new FL method that can significantly mitigate statistical heterogeneity through the depersonalization mechanism. Particularly, we decouple the global and local optimization objectives by alternating stochastic gradient descent, thus reducing the accumulated variance in local update phases to accelerate the FL convergence. Then we analyze the proposed method detailedly to show the proposed method converging at a sublinear speed in the general non-convex setting. Finally, numerical results are conducted with experiments on public datasets to verify the effectiveness of our proposed method.