CoBo: Collaborative Learning via Bilevel Optimization

TL;DR

This paper introduces CoBo, a bilevel optimization framework for collaborative learning that improves client selection and training efficiency, achieving higher accuracy in heterogeneous distributed datasets.

Contribution

The paper proposes a novel bilevel optimization model and an efficient SGD-type algorithm, CoBo, with theoretical convergence guarantees for collaborative learning.

Findings

CoBo surpasses popular personalization algorithms by 9.3% accuracy.

It effectively handles high heterogeneity in distributed datasets.

The algorithm demonstrates scalable and elastic performance.

Abstract

Collaborative learning is an important tool to train multiple clients more effectively by enabling communication among clients. Identifying helpful clients, however, presents challenging and often introduces significant overhead. In this paper, we model client-selection and model-training as two interconnected optimization problems, proposing a novel bilevel optimization problem for collaborative learning. We introduce CoBo, a scalable and elastic, SGD-type alternating optimization algorithm that efficiently addresses these problem with theoretical convergence guarantees. Empirically, CoBo achieves superior performance, surpassing popular personalization algorithms by 9.3% in accuracy on a task with high heterogeneity, involving datasets distributed among 80 clients.