FedDuA: Doubly Adaptive Federated Learning

TL;DR

FedDuA introduces a doubly adaptive federated learning framework that dynamically adjusts learning rates based on data heterogeneity, leading to faster convergence and improved robustness without extra client costs.

Contribution

This work formalizes server optimization via mirror descent and proposes a novel adaptive step-size rule that is minimax optimal, enhancing federated learning efficiency.

Findings

Outperforms baseline methods in various experimental settings.

Achieves faster convergence without additional client communication.

Robust to hyperparameter choices.

Abstract

Federated learning is a distributed learning framework where clients collaboratively train a global model without sharing their raw data. FedAvg is a popular algorithm for federated learning, but it often suffers from slow convergence due to the heterogeneity of local datasets and anisotropy in the parameter space. In this work, we formalize the central server optimization procedure through the lens of mirror descent and propose a novel framework, called FedDuA, which adaptively selects the global learning rate based on both inter-client and coordinate-wise heterogeneity in the local updates. We prove that our proposed doubly adaptive step-size rule is minimax optimal and provide a convergence analysis for convex objectives. Although the proposed method does not require additional communication or computational cost on clients, extensive numerical experiments show that our proposed framework outperforms baselines in various settings and is robust to the choice of hyperparameters.