On the Convergence of FedAvg on Non-IID Data

TL;DR

This paper provides a theoretical analysis of FedAvg's convergence on non-iid data, establishing rates, trade-offs, and conditions for effective federated learning in realistic, heterogeneous environments.

Contribution

It offers the first convergence rate analysis of FedAvg on non-iid data, including partial participation, heterogeneity effects, and necessary learning rate decay conditions.

Findings

Convergence rate of O(1/T) for strongly convex, smooth problems.

Trade-off between communication efficiency and convergence speed.

Heterogeneity slows down convergence, matching empirical observations.

Abstract

Federated learning enables a large amount of edge computing devices to jointly learn a model without data sharing. As a leading algorithm in this setting, Federated Averaging (\texttt{FedAvg}) runs Stochastic Gradient Descent (SGD) in parallel on a small subset of the total devices and averages the sequences only once in a while. Despite its simplicity, it lacks theoretical guarantees under realistic settings. In this paper, we analyze the convergence of \texttt{FedAvg} on non-iid data and establish a convergence rate of $\mathcal{O}(\frac{1}{T})$ for strongly convex and smooth problems, where $T$ is the number of SGDs. Importantly, our bound demonstrates a trade-off between communication-efficiency and convergence rate. As user devices may be disconnected from the server, we relax the assumption of full device participation to partial device participation and study different averaging schemes; low device participation rate can be achieved without severely slowing down the learning. Our results indicate that heterogeneity of data slows down the convergence, which matches empirical observations. Furthermore, we provide a necessary condition for \texttt{FedAvg} on non-iid data: the learning rate $\eta$ must decay, even if full-gradient is used; otherwise, the solution will be $\Omega (\eta)$ away from the optimal.