Efficient Federated Learning against Heterogeneous and Non-stationary Client Unavailability

TL;DR

This paper introduces FedAPM, a federated learning algorithm designed to handle heterogeneous and non-stationary client unavailability with minimal additional memory and computation, ensuring convergence and efficiency.

Contribution

FedAPM is a novel federated learning method that compensates for client unavailability and diffuses updates effectively, even under non-stationary conditions, with theoretical convergence guarantees.

Findings

FedAPM converges to a stationary point in non-convex settings.

Achieves linear speedup similar to standard FedAvg.

Performs well on real-world datasets with dynamic client availability.

Abstract

Addressing intermittent client availability is critical for the real-world deployment of federated learning algorithms. Most prior work either overlooks the potential non-stationarity in the dynamics of client unavailability or requires substantial memory/computation overhead. We study federated learning in the presence of heterogeneous and non-stationary client availability, which may occur when the deployment environments are uncertain, or the clients are mobile. The impacts of heterogeneity and non-stationarity on client unavailability can be significant, as we illustrate using FedAvg, the most widely adopted federated learning algorithm. We propose FedAPM, which includes novel algorithmic structures that (i) compensate for missed computations due to unavailability with only $O(1)$ additional memory and computation with respect to standard FedAvg, and (ii) evenly diffuse local updates within the federated learning system through implicit gossiping, despite being agnostic to non-stationary dynamics. We show that FedAPM converges to a stationary point of even non-convex objectives while achieving the desired linear speedup property. We corroborate our analysis with numerical experiments over diversified client unavailability dynamics on real-world data sets.