A subsampling approach for Bayesian model selection

TL;DR

This paper introduces a subsampling-based Bayesian model selection method that uses stochastic gradient descent to efficiently estimate marginal likelihoods in large datasets, improving scalability.

Contribution

It proposes a novel subsampling approach combined with MCMC for scalable Bayesian model selection in generalized linear models.

Findings

The method achieves comparable accuracy to traditional approaches.

It significantly reduces computational time on large datasets.

Theoretical convergence guarantees are provided.

Abstract

It is common practice to use Laplace approximations to compute marginal likelihoods in Bayesian versions of generalised linear models (GLM). Marginal likelihoods combined with model priors are then used in different search algorithms to compute the posterior marginal probabilities of models and individual covariates. This allows performing Bayesian model selection and model averaging. For large sample sizes, even the Laplace approximation becomes computationally challenging because the optimisation routine involved needs to evaluate the likelihood on the full set of data in multiple iterations. As a consequence, the algorithm is not scalable for large datasets. To address this problem, we suggest using a version of a popular batch stochastic gradient descent (BSGD) algorithm for estimating the marginal likelihood of a GLM by subsampling from the data. We further combine the algorithm with Markov chain Monte Carlo (MCMC) based methods for Bayesian model selection and provide some theoretical results on the convergence of the estimates. Finally, we report results from experiments illustrating the performance of the proposed algorithm.