Decentralized Bayesian Learning over Graphs

TL;DR

This paper introduces a decentralized Bayesian learning algorithm for social networks that enables agents with local data to collaboratively learn a global model, with strong convergence guarantees and applicability to Bayesian neural networks.

Contribution

It presents a Bayesian, peer-to-peer variant of federated learning that handles arbitrary network graphs and provides theoretical convergence analysis and practical training methods.

Findings

Algorithm converges under general network conditions

Provides a closed-form rate of convergence

Enables decentralized training of Bayesian neural networks

Abstract

We propose a decentralized learning algorithm over a general social network. The algorithm leaves the training data distributed on the mobile devices while utilizing a peer to peer model aggregation method. The proposed algorithm allows agents with local data to learn a shared model explaining the global training data in a decentralized fashion. The proposed algorithm can be viewed as a Bayesian and peer-to-peer variant of federated learning in which each agent keeps a "posterior probability distribution" over a global model parameters. The agent update its "posterior" based on 1) the local training data and 2) the asynchronous communication and model aggregation with their 1-hop neighbors. This Bayesian formulation allows for a systematic treatment of model aggregation over any arbitrary connected graph. Furthermore, it provides strong analytic guarantees on converge in the realizable case as well as a closed form characterization of the rate of convergence. We also show that our methodology can be combined with efficient Bayesian inference techniques to train Bayesian neural networks in a decentralized manner. By empirical studies we show that our theoretical analysis can guide the design of network/social interactions and data partitioning to achieve convergence.