LoSAC: An Efficient Local Stochastic Average Control Method for Federated Optimization

TL;DR

LoSAC is a novel federated optimization method that enhances communication efficiency, reduces model divergence, and defends against information leakage, demonstrating superior performance over existing algorithms.

Contribution

LoSAC introduces a local gradient estimation update technique that improves efficiency, convergence, and privacy in federated learning.

Findings

Over 100% improvement in communication efficiency.

Effective mitigation of model divergence.

Enhanced defense against gradient leakage techniques.

Abstract

Federated optimization (FedOpt), which targets at collaboratively training a learning model across a large number of distributed clients, is vital for federated learning. The primary concerns in FedOpt can be attributed to the model divergence and communication efficiency, which significantly affect the performance. In this paper, we propose a new method, i.e., LoSAC, to learn from heterogeneous distributed data more efficiently. Its key algorithmic insight is to locally update the estimate for the global full gradient after {each} regular local model update. Thus, LoSAC can keep clients' information refreshed in a more compact way. In particular, we have studied the convergence result for LoSAC. Besides, the bonus of LoSAC is the ability to defend the information leakage from the recent technique Deep Leakage Gradients (DLG). Finally, experiments have verified the superiority of LoSAC comparing with state-of-the-art FedOpt algorithms. Specifically, LoSAC significantly improves communication efficiency by more than $100\%$ on average, mitigates the model divergence problem and equips with the defense ability against DLG.