Asynchronous Federated Stochastic Optimization for Heterogeneous Objectives Under Arbitrary Delays

TL;DR

This paper introduces AREA, an asynchronous federated learning algorithm that corrects bias from heterogeneous delays without client sampling, achieving optimal convergence rates and robustness to outliers.

Contribution

The paper proposes AREA, a novel asynchronous stochastic gradient method for federated learning that handles arbitrary delays and heterogeneity without client sampling or prior delay knowledge.

Findings

Achieves optimal convergence rates in convex and strongly convex regimes.

Supports larger step sizes for faster convergence.

Rates scale with average client update frequency, enhancing robustness.

Abstract

Federated learning (FL) was recently proposed to securely train models with data held over multiple locations (``clients'') under the coordination of a central server. Prolonged training times caused by slow clients may hinder the performance of FL; while asynchronous communication is a promising solution, highly heterogeneous client response times under non-IID local data may introduce significant bias to the global model, particularly in client-driven setups where sampling is infeasible. To address this issue, we propose \underline{A}synch\underline{R}onous \underline{E}xact \underline{A}veraging (\textsc{AREA}), a stochastic (sub)gradient method that leverages asynchrony for scalability and uses client-side memory to correct the bias induced by uneven participation, without client sampling or prior knowledge of client latencies. \textsc{AREA} communicates model residuals rather than gradient estimates, reducing exposure to gradient inversion, and is compatible with secure aggregation. Under standard assumptions and unbounded, heterogeneous delays with finite mean, AREA achieves optimal convergence rates: $\mathcal{O}(1/K)$ in the strongly convex, smooth regime and $\mathcal{O}(1/\sqrt{K})$ in the convex, nonsmooth regime. For strongly convex, smooth objectives, we demonstrate theoretically and empirically that AREA accommodates larger step sizes than existing methods, enabling fast convergence without adversely impacting model generalization. In the convex, nonsmooth setting, to our knowledge we are the first to obtain rates that scale with the average client update frequency rather than the minimum or maximum, indicating increased robustness to outliers.