Distributed Federated Learning by Alternating Periods of Training

TL;DR

This paper introduces a distributed federated learning framework with multiple servers, enabling scalable and fault-tolerant model training through alternating local client updates and inter-server communication, supported by theoretical convergence guarantees.

Contribution

It proposes a novel distributed federated learning algorithm that replaces a central server with multiple interconnected servers, enhancing scalability and fault tolerance.

Findings

The DFL algorithm converges to a common model across servers.

The framework maintains federated learning structure with decentralized architecture.

Numerical simulations validate theoretical convergence results.

Abstract

Federated learning is a privacy-focused approach towards machine learning where models are trained on client devices with locally available data and aggregated at a central server. However, the dependence on a single central server is challenging in the case of a large number of clients and even poses the risk of a single point of failure. To address these critical limitations of scalability and fault-tolerance, we present a distributed approach to federated learning comprising multiple servers with inter-server communication capabilities. While providing a fully decentralized approach, the designed framework retains the core federated learning structure where each server is associated with a disjoint set of clients with server-client communication capabilities. We propose a novel DFL (Distributed Federated Learning) algorithm which uses alternating periods of local training on the client data followed by global training among servers. We show that the DFL algorithm, under a suitable choice of parameters, ensures that all the servers converge to a common model value within a small tolerance of the ideal model, thus exhibiting effective integration of local and global training models. Finally, we illustrate our theoretical claims through numerical simulations.