Towards Practical Overlay Networks for Decentralized Federated Learning

TL;DR

This paper introduces FedLay, a decentralized overlay network for federated learning that enables fast, accurate training with low communication costs and robust maintenance under node changes.

Contribution

FedLay is the first decentralized protocol for constructing and maintaining near-random regular topologies in DFL, improving training speed and accuracy.

Findings

FedLay achieves faster convergence than existing solutions.

It maintains high accuracy with low communication overhead.

The topology is resilient to node joins and failures.

Abstract

Decentralized federated learning (DFL) uses peer-to-peer communication to avoid the single point of failure problem in federated learning and has been considered an attractive solution for machine learning tasks on distributed devices. We provide the first solution to a fundamental network problem of DFL: what overlay network should DFL use to achieve fast training of highly accurate models, low communication, and decentralized construction and maintenance? Overlay topologies of DFL have been investigated, but no existing DFL topology includes decentralized protocols for network construction and topology maintenance. Without these protocols, DFL cannot run in practice. This work presents an overlay network, called FedLay, which provides fast training and low communication cost for practical DFL. FedLay is the first solution for constructing near-random regular topologies in a decentralized manner and maintaining the topologies under node joins and failures. Experiments based on prototype implementation and simulations show that FedLay achieves the fastest model convergence and highest accuracy on real datasets compared to existing DFL solutions while incurring small communication costs and being resilient to node joins and failures.