Parallelizing MCMC with Machine Learning Classifier and Its Criterion Based on Kullback-Leibler Divergence

TL;DR

This paper introduces a novel parallel MCMC method using machine learning classifiers and a KL divergence-based criterion to efficiently approximate full posterior distributions from partitioned data, enhancing computational speed.

Contribution

It proposes a new approach combining classifiers and KL divergence to parallelize MCMC without full posterior calculations, improving scalability for large datasets.

Findings

Validated through simulation studies

Effective in identifying and extracting parallel MCMC draws

Reduces computational complexity in Bayesian analysis

Abstract

In the era of Big Data, Markov chain Monte Carlo (MCMC) methods, which are currently essential for Bayesian estimation, face significant computational challenges owing to their sequential nature. To achieve a faster and more effective parallel computation, we emphasize the critical role of the overlapped area of the posterior distributions based on partitioned data, which we term the reconstructable area. We propose a method that utilizes machine learning classifiers to effectively identify and extract MCMC draws obtained by parallel computations from the area based on posteriors based on partitioned sub-datasets, approximating the target posterior distribution based on the full dataset. This study also develops a Kullback-Leibler (KL) divergence-based criterion. It does not require calculating the full-posterior density and can be calculated using only information from the sub-posterior densities, which are generally obtained after implementing MCMC. This simplifies the hyperparameter tuning in training classifiers. The simulation studies validated the efficacy of the proposed method. This approach contributes to ongoing research on parallelizing MCMC methods and may offer insights for future developments in Bayesian computation for large-scale data analyses.