Linear Speedup in Personalized Collaborative Learning

TL;DR

This paper analyzes personalized collaborative learning, providing theoretical guarantees for convergence and demonstrating conditions for linear speedup with multiple related tasks, supported by empirical results.

Contribution

It formalizes the personalized collaborative learning problem, introduces a bias correction method, and establishes conditions for linear speedup in convergence.

Findings

Convergence guarantees for weighted gradient averaging and bias correction methods.

Conditions under which linear speedup is achievable with multiple tasks.

Empirical validation of theoretical insights.

Abstract

Collaborative training can improve the accuracy of a model for a user by trading off the model's bias (introduced by using data from other users who are potentially different) against its variance (due to the limited amount of data on any single user). In this work, we formalize the personalized collaborative learning problem as a stochastic optimization of a task 0 while giving access to N related but different tasks 1,..., N. We provide convergence guarantees for two algorithms in this setting -- a popular collaboration method known as weighted gradient averaging, and a novel bias correction method -- and explore conditions under which we can achieve linear speedup w.r.t. the number of auxiliary tasks N. Further, we also empirically study their performance confirming our theoretical insights.