Scalable MCMC for Large Data Problems using Data Subsampling and the Difference Estimator

TL;DR

This paper introduces a scalable MCMC algorithm that uses data subsampling and the difference estimator to efficiently approximate the likelihood, enabling faster Bayesian inference on large datasets.

Contribution

The authors develop a generic MCMC method leveraging the difference estimator for accurate likelihood estimation from small data subsamples, reducing computational complexity.

Findings

Significant speed-up over standard MCMC on large datasets.

Accurate posterior approximation within O(m^{-1/2}) of the true posterior.

Effective application to logistic regression for bankruptcy prediction.

Abstract

We propose a generic Markov Chain Monte Carlo (MCMC) algorithm to speed up computations for datasets with many observations. A key feature of our approach is the use of the highly efficient difference estimator from the survey sampling literature to estimate the log-likelihood accurately using only a small fraction of the data. Our algorithm improves on the $O(n)$ complexity of regular MCMC by operating over local data clusters instead of the full sample when computing the likelihood. The likelihood estimate is used in a Pseudo-marginal framework to sample from a perturbed posterior which is within $O(m^{-1/2})$ of the true posterior, where $m$ is the subsample size. The method is applied to a logistic regression model to predict firm bankruptcy for a large data set. We document a significant speed up in comparison to the standard MCMC on the full dataset.