Decentralized Machine Learning with Centralized Performance Guarantees via Gibbs Algorithms

TL;DR

This paper demonstrates that decentralized learning can match centralized performance by sharing Gibbs measures instead of data, using a specific ERM-RER framework and communication protocol.

Contribution

It introduces a method where decentralized clients share Gibbs measures to achieve centralized performance without data sharing, with a principled regularization scaling.

Findings

Decentralized Gibbs measures can replicate centralized ERM-RER performance.

Sharing local Gibbs measures suffices for optimal collaborative learning.

Regularization factors must scale with local sample sizes for success.

Abstract

In this paper, it is shown, for the first time, that centralized performance is achievable in decentralized learning without sharing the local datasets. Specifically, when clients adopt an empirical risk minimization with relative-entropy regularization (ERM-RER) learning framework and a forward-backward communication between clients is established, it suffices to share the locally obtained Gibbs measures to achieve the same performance as that of a centralized ERM-RER with access to all the datasets. The core idea is that the Gibbs measure produced by client~$k$ is used, as reference measure, by client~$k+1$. This effectively establishes a principled way to encode prior information through a reference measure. In particular, achieving centralized performance in the decentralized setting requires a specific scaling of the regularization factors with the local sample sizes. Overall, this result opens the door to novel decentralized learning paradigms that shift the collaboration strategy from sharing data to sharing the local inductive bias via the reference measures over the set of models.