FADAS: Towards Federated Adaptive Asynchronous Optimization

TL;DR

This paper introduces FADAS, a novel federated learning optimization method that incorporates asynchronous updates and delay-adaptive strategies, improving efficiency and convergence in large-scale, privacy-preserving machine learning scenarios.

Contribution

FADAS is the first to integrate asynchronous updates with adaptive federated optimization, providing provable guarantees and enhanced resilience to delays.

Findings

FADAS outperforms existing asynchronous FL baselines in experiments.

The convergence rate of FADAS is rigorously established.

Delay-adaptive strategies improve robustness against asynchronous delays.

Abstract

Federated learning (FL) has emerged as a widely adopted training paradigm for privacy-preserving machine learning. While the SGD-based FL algorithms have demonstrated considerable success in the past, there is a growing trend towards adopting adaptive federated optimization methods, particularly for training large-scale models. However, the conventional synchronous aggregation design poses a significant challenge to the practical deployment of those adaptive federated optimization methods, particularly in the presence of straggler clients. To fill this research gap, this paper introduces federated adaptive asynchronous optimization, named FADAS, a novel method that incorporates asynchronous updates into adaptive federated optimization with provable guarantees. To further enhance the efficiency and resilience of our proposed method in scenarios with significant asynchronous delays, we also extend FADAS with a delay-adaptive learning adjustment strategy. We rigorously establish the convergence rate of the proposed algorithms and empirical results demonstrate the superior performance of FADAS over other asynchronous FL baselines.