Overlay-based Decentralized Federated Learning in Bandwidth-limited Networks

TL;DR

This paper proposes an overlay-based decentralized federated learning approach that optimizes communication scheduling in bandwidth-limited networks, significantly improving training speed without relying on network cooperation.

Contribution

It introduces a novel method leveraging network tomography to jointly optimize communication demands and schedules in DFL over general bandwidth-limited networks.

Findings

Significant acceleration of DFL training time compared to existing methods.

Effective joint optimization of communication demands and schedules.

No need for explicit cooperation from underlying networks.

Abstract

The emerging machine learning paradigm of decentralized federated learning (DFL) has the promise of greatly boosting the deployment of artificial intelligence (AI) by directly learning across distributed agents without centralized coordination. Despite significant efforts on improving the communication efficiency of DFL, most existing solutions were based on the simplistic assumption that neighboring agents are physically adjacent in the underlying communication network, which fails to correctly capture the communication cost when learning over a general bandwidth-limited network, as encountered in many edge networks. In this work, we address this gap by leveraging recent advances in network tomography to jointly design the communication demands and the communication schedule for overlay-based DFL in bandwidth-limited networks without requiring explicit cooperation from the underlying network. By carefully analyzing the structure of our problem, we decompose it into a series of optimization problems that can each be solved efficiently, to collectively minimize the total training time. Extensive data-driven simulations show that our solution can significantly accelerate DFL in comparison with state-of-the-art designs.