Adaptive collaboration for online personalized distributed learning with heterogeneous clients

TL;DR

This paper introduces a dynamic, gradient-based collaboration method for online personalized decentralized learning, enabling heterogeneous clients to select relevant peers, reduce variance, and improve training efficiency.

Contribution

It proposes a novel gradient-based peer selection criterion for decentralized learning, with theoretical analysis and two practical collaboration algorithms demonstrating improved performance.

Findings

The proposed method reduces gradient variance effectively.

One variant maintains the optimality of the All-for-one algorithm.

Experimental results confirm improved training speed and personalization.

Abstract

We study the problem of online personalized decentralized learning with $N$ statistically heterogeneous clients collaborating to accelerate local training. An important challenge in this setting is to select relevant collaborators to reduce gradient variance while mitigating the introduced bias. To tackle this, we introduce a gradient-based collaboration criterion, allowing each client to dynamically select peers with similar gradients during the optimization process. Our criterion is motivated by a refined and more general theoretical analysis of the All-for-one algorithm, proved to be optimal in Even et al. (2022) for an oracle collaboration scheme. We derive excess loss upper-bounds for smooth objective functions, being either strongly convex, non-convex, or satisfying the Polyak-Lojasiewicz condition; our analysis reveals that the algorithm acts as a variance reduction method where the speed-up depends on a sufficient variance. We put forward two collaboration methods instantiating the proposed general schema; and we show that one variant preserves the optimality of All-for-one. We validate our results with experiments on synthetic and real datasets.