Asynchronous Multi-Server Federated Learning for Geo-Distributed Clients

TL;DR

This paper introduces the first fully asynchronous multi-server federated learning system that enhances scalability and reduces training time by maintaining continuous activity of servers and clients in geo-distributed environments.

Contribution

It proposes a novel multi-server asynchronous federated learning architecture where servers update each other asynchronously, improving efficiency over existing methods.

Findings

Achieves similar or higher accuracy compared to baselines.

Reduces training time by 61% in geo-distributed settings.

Maintains continuous server and client activity for better scalability.

Abstract

Federated learning (FL) systems enable multiple clients to train a machine learning model iteratively through synchronously exchanging the intermediate model weights with a single server. The scalability of such FL systems can be limited by two factors: server idle time due to synchronous communication and the risk of a single server becoming the bottleneck. In this paper, we propose a new FL architecture, to our knowledge, the first multi-server FL system that is entirely asynchronous, and therefore addresses these two limitations simultaneously. Our solution keeps both servers and clients continuously active. As in previous multi-server methods, clients interact solely with their nearest server, ensuring efficient update integration into the model. Differently, however, servers also periodically update each other asynchronously, and never postpone interactions with clients. We compare our solution to three representative baselines - FedAvg, FedAsync and HierFAVG - on the MNIST and CIFAR-10 image classification datasets and on the WikiText-2 language modeling dataset. Our solution converges to similar or higher accuracy levels than previous baselines and requires 61% less time to do so in geo-distributed settings.