A Novel Coded Computing Approach for Distributed Multi-Task Learning

TL;DR

This paper introduces a new coded computing scheme for distributed multi-task learning that significantly reduces communication costs and achieves theoretical optimality, applicable to both homogeneous and heterogeneous environments.

Contribution

It proposes a novel coded scheme for DMTL that minimizes communication overhead and is optimal under mild conditions, extending to heterogeneous data placements.

Findings

Achieves the theoretical lower bound for communication cost.

Effective in both homogeneous and heterogeneous computing environments.

Extensible to distributed linear separable computation problems.

Abstract

Distributed multi-task learning (DMTL) effectively improves model generalization performance through the collaborative training of multiple related models. However, in large-scale learning scenarios, communication bottlenecks severely limit practical system performance. In this paper, we investigate the communication bottleneck within a typical DMTL system that employs non-linear global updates. This system involves distributed workers, assisted by a central server, who collaboratively learn distinct models derived from a non-linear aggregation of their local model parameters. We first characterize the communication process as a matrix decomposition problem. It transforms workers' data storage constraints into structural characteristics of the uplink encoding matrix, and worker data retrieval demands into Maximum Distance Separable (MDS) properties of the downlink encoding matrix. Building on this, we propose a novel coded DTML scheme that can greatly reduce the communication cost of the DTML with heterogeneous data placement. Theoretical analysis demonstrates that the proposed scheme achieves the theoretical lower bound for communication overhead under mild conditions. Remarkably, this optimality holds for both traditional homogeneous computing environments and various heterogeneous scenarios. Furthermore, our scheme is extensible to a distributed linearly separable computation problem where the target function involves multiple linear combinations of local update values. This indicates that our scheme offers a new way of tackling heterogeneous data placement challenges in various distributed applications.