Efficient Adaptive Federated Optimization

TL;DR

This paper introduces $FedAda^2$ and $FedAda^2$++ algorithms for scalable, resource-efficient adaptive federated learning, achieving optimal convergence rates while reducing communication and memory overhead.

Contribution

The paper proposes novel adaptive federated optimization algorithms that improve scalability and resource efficiency without sacrificing convergence guarantees.

Findings

$FedAda^2$ and $FedAda^2$++ match the convergence rates of more resource-intensive methods.

Empirical results show improved performance on image and text datasets.

Resource-efficient adaptivity enhances federated learning scalability.

Abstract

Adaptive optimization is critical in federated learning, where enabling adaptivity on both the server and client sides has proven essential for achieving optimal performance. However, the scalability of such jointly adaptive systems is often hindered by resource limitations in communication and memory. In this paper, we introduce a class of efficient adaptive algorithms, named $FedAda^2$ and its enhanced version $FedAda^2$++, designed specifically for large-scale, cross-device federated environments. $FedAda^2$ optimizes communication efficiency by avoiding the transfer of preconditioners between the server and clients. Additionally, $FedAda^2$++ extends this approach by incorporating memory-efficient adaptive optimizers on the client side, further reducing on-device memory usage. Theoretically, we demonstrate that $FedAda^2$ and $FedAda^2$++ achieve the same convergence rates for general, non-convex objectives as its more resource-intensive counterparts that directly integrate joint adaptivity. Extensive empirical evaluations on image and text datasets demonstrate both the advantages of joint adaptivity and the effectiveness of $FedAda^2$/$FedAda^2$++.