Coded Distributed Computing for Hierarchical Multi-task Learning

TL;DR

This paper introduces a coded hierarchical multi-task learning scheme that leverages network topology and coding to significantly reduce communication loads and training time in distributed multi-task learning systems.

Contribution

It proposes a novel coded hierarchical MTL scheme that exploits connection topology and provides theoretical bounds, achieving near-optimal communication efficiency.

Findings

Reduces communication loads in uplink and downlink transmissions.

Achieves near-optimal bounds within the minimum number of connected users.

Decreases overall training time by 17% to 26% in experiments.

Abstract

In this paper, we consider a hierarchical distributed multi-task learning (MTL) system where distributed users wish to jointly learn different models orchestrated by a central server with the help of a layer of multiple relays. Since the users need to download different learning models in the downlink transmission, the distributed MTL suffers more severely from the communication bottleneck compared to the single-task learning system. To address this issue, we propose a coded hierarchical MTL scheme that exploits the connection topology and introduces coding techniques to reduce communication loads. It is shown that the proposed scheme can significantly reduce the communication loads both in the uplink and downlink transmissions between relays and the server. Moreover, we provide information-theoretic lower bounds on the optimal uplink and downlink communication loads, and prove that the gaps between achievable upper bounds and lower bounds are within the minimum number of connected users among all relays. In particular, when the network connection topology can be delicately designed, the proposed scheme can achieve the information-theoretic optimal communication loads. Experiments on real datasets show that our proposed scheme can reduce the overall training time by 17% $\sim$ 26% compared to the conventional uncoded scheme.