Collaborative Learning over Wireless Networks: An Introductory Overview

TL;DR

This paper provides an overview of collaborative machine learning over wireless networks, emphasizing the challenges of communication bottlenecks, device heterogeneity, and the need for tailored distributed algorithms at the network edge.

Contribution

It offers an introductory survey of distributed optimization algorithms for wireless edge devices, highlighting practical challenges and considerations for effective collaborative learning.

Findings

Communication bottlenecks limit learning speed gains.

Device heterogeneity affects distributed training efficiency.

Time-varying network conditions require adaptive algorithms.

Abstract

In this chapter, we will mainly focus on collaborative training across wireless devices. Training a ML model is equivalent to solving an optimization problem, and many distributed optimization algorithms have been developed over the last decades. These distributed ML algorithms provide data locality; that is, a joint model can be trained collaboratively while the data available at each participating device remains local. This addresses, to some extend, the privacy concern. They also provide computational scalability as they allow exploiting computational resources distributed across many edge devices. However, in practice, this does not directly lead to a linear gain in the overall learning speed with the number of devices. This is partly due to the communication bottleneck limiting the overall computation speed. Additionally, wireless devices are highly heterogeneous in their computational capabilities, and both their computation speed and communication rate can be highly time-varying due to physical factors. Therefore, distributed learning algorithms, particularly those to be implemented at the wireless network edge, must be carefully designed taking into account the impact of time-varying communication network as well as the heterogeneous and stochastic computation capabilities of devices.