Communication Efficient Distributed Learning over Wireless Channels

TL;DR

This paper introduces a hierarchical distributed learning framework that reduces communication costs over wireless channels by using low-dimensional embeddings and max-pooling, achieving near-equivalent accuracy with less data exchange.

Contribution

It proposes a novel hierarchical framework with max-pooling and an opportunistic protocol for efficient wireless communication in distributed learning.

Findings

Achieves similar accuracy to raw data concatenation methods.

Reduces communication load independently of the number of workers.

Demonstrates effectiveness through simulation experiments.

Abstract

Vertical distributed learning exploits the local features collected by multiple learning workers to form a better global model. However, the exchange of data between the workers and the model aggregator for parameter training incurs a heavy communication burden, especially when the learning system is built upon capacity-constrained wireless networks. In this paper, we propose a novel hierarchical distributed learning framework, where each worker separately learns a low-dimensional embedding of their local observed data. Then, they perform communication efficient distributed max-pooling for efficiently transmitting the synthesized input to the aggregator. For data exchange over a shared wireless channel, we propose an opportunistic carrier sensing-based protocol to implement the max-pooling operation for the output data from all the learning workers. Our simulation experiments show that the proposed learning framework is able to achieve almost the same model accuracy as the learning model using the concatenation of all the raw outputs from the learning workers, while requiring a communication load that is independent of the number of workers.