On the Communication Latency of Wireless Decentralized Learning

TL;DR

This paper analyzes how communication delay scales in wireless decentralized learning networks, revealing that delay depends on network size and communication range, with implications for designing efficient distributed algorithms.

Contribution

It provides a theoretical analysis of communication latency in wireless decentralized learning, linking delay to network parameters using tools from information and geometric graph theory.

Findings

Communication delay scales as 5B9(rac{n^{2-3eta}}{eta\log n}) for a network of n nodes.

Delay increases with the number of nodes and the communication threshold distance.

The analysis guides the design of scalable decentralized learning protocols.

Abstract

We consider a wireless network comprising $n$ nodes located within a circular area of radius $R$, which are participating in a decentralized learning algorithm to optimize a global objective function using their local datasets. To enable gradient exchanges across the network, we assume each node communicates only with a set of neighboring nodes, which are within a distance $R n^{-\beta}$ of itself, where $\beta\in(0,\frac{1}{2})$. We use tools from network information theory and random geometric graph theory to show that the communication delay for a single round of exchanging gradients on all the links throughout the network scales as $\mathcal{O}\left(\frac{n^{2-3\beta}}{\beta\log n}\right)$, increasing (at different rates) with both the number of nodes and the gradient exchange threshold distance.