An Algorithm for Distributed Bayesian Inference in Generalized Linear Models

TL;DR

This paper introduces a scalable distributed Bayesian inference algorithm for generalized linear models that efficiently handles massive datasets by dividing data, parallelizing computations, and combining results, maintaining statistical accuracy.

Contribution

It develops a novel divide-and-conquer Bayesian inference method using powered likelihoods, with theoretical guarantees and practical effectiveness demonstrated on linear and logistic regressions.

Findings

Comparable accuracy to state-of-the-art algorithms

Significant computational efficiency gains

Asymptotic optimality under regularity conditions

Abstract

Monte Carlo algorithms, such as Markov chain Monte Carlo (MCMC) and Hamiltonian Monte Carlo (HMC), are routinely used for Bayesian inference in generalized linear models; however, these algorithms are prohibitively slow in massive data settings because they require multiple passes through the full data in every iteration. Addressing this problem, we develop a scalable extension of these algorithms using the divide-and-conquer (D&C) technique that divides the data into a sufficiently large number of subsets, draws parameters in parallel on the subsets using a \textit{powered} likelihood, and produces Monte Carlo draws of the parameter by combining parameter draws obtained from each subset. These combined parameter draws play the role of draws from the original sampling algorithm. Our main contributions are two-fold. First, we demonstrate through diverse simulated and real data analyses that our distributed algorithm is comparable to the current state-of-the-art D&C algorithm in terms of statistical accuracy and computational efficiency. Second, providing theoretical support for our empirical observations, we identify regularity assumptions under which the proposed algorithm leads to asymptotically optimal inference. We illustrate our methodology through normal linear and logistic regressions, where parts of our D&C algorithm are analytically tractable.