FedHyper: A Universal and Robust Learning Rate Scheduler for Federated Learning with Hypergradient Descent

TL;DR

FedHyper introduces a hypergradient-based learning rate scheduler for federated learning, enabling faster convergence and higher accuracy with reduced need for manual hyperparameter tuning.

Contribution

It proposes a universal, robust learning rate adaptation algorithm for federated learning that improves convergence speed and accuracy, addressing hyperparameter tuning challenges.

Findings

Converges 1.1-3x faster than FedAvg.

Achieves up to 15% higher accuracy under suboptimal initial rates.

Demonstrates robustness across vision and language benchmarks.

Abstract

The theoretical landscape of federated learning (FL) undergoes rapid evolution, but its practical application encounters a series of intricate challenges, and hyperparameter optimization is one of these critical challenges. Amongst the diverse adjustments in hyperparameters, the adaptation of the learning rate emerges as a crucial component, holding the promise of significantly enhancing the efficacy of FL systems. In response to this critical need, this paper presents FedHyper, a novel hypergradient-based learning rate adaptation algorithm specifically designed for FL. FedHyper serves as a universal learning rate scheduler that can adapt both global and local rates as the training progresses. In addition, FedHyper not only showcases unparalleled robustness to a spectrum of initial learning rate configurations but also significantly alleviates the necessity for laborious empirical learning rate adjustments. We provide a comprehensive theoretical analysis of FedHyper's convergence rate and conduct extensive experiments on vision and language benchmark datasets. The results demonstrate that FEDHYPER consistently converges 1.1-3x faster than FedAvg and the competing baselines while achieving superior final accuracy. Moreover, FedHyper catalyzes a remarkable surge in accuracy, augmenting it by up to 15% compared to FedAvg under suboptimal initial learning rate settings.