Reversible Jump MCMC With No Regrets: Bayesian Variable Selection Using Mixtures of Mutually Singular Distributions

TL;DR

This paper introduces Mixtures of Mutually Singular (MoMS) distributions as a transparent and effective alternative to reversible jump MCMC for Bayesian variable selection, simplifying implementation while maintaining accuracy.

Contribution

The authors propose MoMS distributions to represent models within a fixed-dimensional space, reproducing spike-and-slab interpretation and matching RJMCMC in acceptance probability.

Findings

MoMS achieves comparable or better effective sample size than RJMCMC.

Both methods recover posterior inclusion probabilities matching full enumeration.

MoMS simplifies Bayesian variable selection within standard MCMC frameworks.

Abstract

Bayesian variable selection requires sampling from a posterior distribution that combines discrete model indicators with continuously varying parameters, a challenge often addressed through reversible jump Markov chain Monte Carlo (RJMCMC). Despite its generality, RJMCMC is widely regarded as difficult to design and implement correctly. We present mixtures of mutually singular (MoMS) distributions as a transparent alternative in which competing models are represented within a single fixed-dimensional parameter space partitioned into mutually singular subspaces. We show that this formulation reproduces the exact spike-and-slab interpretation of Bayesian variable selection and that, under appropriate constructions, MoMS and RJMCMC share the same Metropolis--Hastings acceptance probability. On a benchmark dataset with ten predictors, both methods recover posterior inclusion probabilities that match full enumeration, while MoMS achieves comparable or superior effective sample size per second relative to a carefully engineered RJMCMC scheme. We further illustrate the approach in a mixed-effects logistic regression for a sleep-and-memory experiment and in factor-loading selection for a multidimensional generalized partial credit model. Together, these results show that Bayesian variable selection can be carried out within standard fixed-dimensional Markov chain Monte Carlo methodology -- without regret.