Beyond Centralization: Provable Communication Efficient Decentralized Multi-Task Learning

TL;DR

This paper introduces a communication-efficient decentralized multi-task learning method that accurately recovers shared low-rank features across distributed data sources, outperforming centralized approaches in certain regimes.

Contribution

It proposes a novel alternating gradient algorithm with provable guarantees, reducing communication complexity independently of accuracy, and provides comprehensive theoretical and empirical analysis.

Findings

Communication complexity is independent of target accuracy.

Decentralized learning can outperform centralized federated methods.

The method is validated across various network topologies.

Abstract

Representation learning is a widely adopted framework for learning in data-scarce environments, aiming to extract common features from related tasks. While centralized approaches have been extensively studied, decentralized methods remain largely underexplored. We study decentralized multi-task representation learning in which the features share a low-rank structure. We consider multiple tasks, each with a finite number of data samples, where the observations follow a linear model with task-specific parameters. In the decentralized setting, task data are distributed across multiple nodes, and information exchange between nodes is constrained by a communication network. The goal is to recover the underlying feature matrix whose rank is much smaller than both the parameter dimension and the number of tasks. We propose a new alternating projected gradient and minimization algorithm with provable accuracy guarantees. We provide comprehensive characterizations of the time, communication, and sample complexities. Importantly, the communication complexity is independent of the target accuracy, which significantly reduces communication cost compared to prior methods. Numerical simulations validate the theoretical analysis across different dimensions and network topologies, and demonstrate regimes in which decentralized learning outperforms centralized federated approaches.