Personalized Decentralized Multi-Task Learning Over Dynamic Communication Graphs

TL;DR

This paper introduces a decentralized multi-task learning algorithm that dynamically adjusts task connections based on exchanged gradients, improving performance and convergence speed in heterogeneous data environments.

Contribution

It proposes a novel method that automatically detects task correlations and adapts the communication graph accordingly, enhancing decentralized multi-task learning.

Findings

Successfully detects positive and negative task correlations.

Achieves faster training convergence on CelebA dataset.

Outperforms fully-connected networks in training speed.

Abstract

Decentralized and federated learning algorithms face data heterogeneity as one of the biggest challenges, especially when users want to learn a specific task. Even when personalized headers are used concatenated to a shared network (PF-MTL), aggregating all the networks with a decentralized algorithm can result in performance degradation as a result of heterogeneity in the data. Our algorithm uses exchanged gradients to calculate the correlations among tasks automatically, and dynamically adjusts the communication graph to connect mutually beneficial tasks and isolate those that may negatively impact each other. This algorithm improves the learning performance and leads to faster convergence compared to the case where all clients are connected to each other regardless of their correlations. We conduct experiments on a synthetic Gaussian dataset and a large-scale celebrity attributes (CelebA) dataset. The experiment with the synthetic data illustrates that our proposed method is capable of detecting tasks that are positively and negatively correlated. Moreover, the results of the experiments with CelebA demonstrate that the proposed method may produce significantly faster training results than fully-connected networks.