Divide-and-Conquer MCMC for Multivariate Binary Data

TL;DR

This paper introduces a divide-and-conquer MCMC method tailored for multivariate probit models to efficiently analyze large-scale medical claims data, enabling insights into correlations among multiple binary health outcomes.

Contribution

It extends divide-and-conquer MCMC algorithms to multivariate probit models, improving scalability and accuracy over variational inference for large medical datasets.

Findings

Identified meaningful groupings of chronic conditions.

Assessed urban-rural health disparities.

Demonstrated computational efficiency on large datasets.

Abstract

The analysis of large scale medical claims data has the potential to improve quality of care by generating insights which can be used to create tailored medical programs. In particular, the multivariate probit model can be used to investigate the correlation between multiple binary responses of interest in such data, e.g. the presence of multiple chronic conditions. Bayesian modeling is well suited to such analyses because of the automatic uncertainty quantification provided by the posterior distribution. A complicating factor is that large medical claims datasets often do not fit in memory, which renders the estimation of the posterior using traditional Markov Chain Monte Carlo (MCMC) methods computationally infeasible. To address this challenge, we extend existing divide-and-conquer MCMC algorithms to the multivariate probit model, demonstrating, via simulation, that they should be preferred over mean-field variational inference when the estimation of the latent correlation structure between binary responses is of primary interest. We apply this algorithm to a large database of de-identified Medicare Advantage claims from a single large US health insurance provider, where we find medically meaningful groupings of common chronic conditions and asses the impact of the urban-rural health gap by identifying underutilized provider specialties in rural areas.