DePRL: Achieving Linear Convergence Speedup in Personalized Decentralized Learning with Shared Representations

TL;DR

DePRL introduces a decentralized personalized learning algorithm that leverages shared representations to achieve linear convergence speedup, effectively handling data heterogeneity without relying on a central server.

Contribution

The paper presents the first provable linear convergence speedup in decentralized learning using shared representations for non-linear models.

Findings

DePRL outperforms existing methods in heterogeneous data environments.

Theoretical proof of linear convergence speedup with increasing number of workers.

Experimental results validate the effectiveness of DePRL in practical scenarios.

Abstract

Decentralized learning has emerged as an alternative method to the popular parameter-server framework which suffers from high communication burden, single-point failure and scalability issues due to the need of a central server. However, most existing works focus on a single shared model for all workers regardless of the data heterogeneity problem, rendering the resulting model performing poorly on individual workers. In this work, we propose a novel personalized decentralized learning algorithm named DePRL via shared representations. Our algorithm relies on ideas from representation learning theory to learn a low-dimensional global representation collaboratively among all workers in a fully decentralized manner, and a user-specific low-dimensional local head leading to a personalized solution for each worker. We show that DePRL achieves, for the first time, a provable linear speedup for convergence with general non-linear representations (i.e., the convergence rate is improved linearly with respect to the number of workers). Experimental results support our theoretical findings showing the superiority of our method in data heterogeneous environments.