Efficient Federated Learning via Local Adaptive Amended Optimizer with Linear Speedup

TL;DR

This paper introduces FedLADA, a novel federated learning optimizer that combines global gradient estimation with local adaptive correction, achieving faster convergence and higher accuracy with fewer communication rounds.

Contribution

The paper proposes FedLADA, a momentum-based adaptive optimizer with a local amendment technique, providing theoretical convergence guarantees and improved empirical performance in federated learning.

Findings

FedLADA achieves linear speedup in convergence rate.

Reduces communication rounds significantly.

Outperforms baseline methods in accuracy on real-world datasets.

Abstract

Adaptive optimization has achieved notable success for distributed learning while extending adaptive optimizer to federated Learning (FL) suffers from severe inefficiency, including (i) rugged convergence due to inaccurate gradient estimation in global adaptive optimizer; (ii) client drifts exacerbated by local over-fitting with the local adaptive optimizer. In this work, we propose a novel momentum-based algorithm via utilizing the global gradient descent and locally adaptive amended optimizer to tackle these difficulties. Specifically, we incorporate a locally amended technique to the adaptive optimizer, named Federated Local ADaptive Amended optimizer (\textit{FedLADA}), which estimates the global average offset in the previous communication round and corrects the local offset through a momentum-like term to further improve the empirical training speed and mitigate the heterogeneous over-fitting. Theoretically, we establish the convergence rate of \textit{FedLADA} with a linear speedup property on the non-convex case under the partial participation settings. Moreover, we conduct extensive experiments on the real-world dataset to demonstrate the efficacy of our proposed \textit{FedLADA}, which could greatly reduce the communication rounds and achieves higher accuracy than several baselines.