Smoothing Out Sticking Points: Sampling from Discrete-Continuous Mixtures with Dynamical Monte Carlo by Mapping Discrete Mass into a Latent Universe

TL;DR

This paper introduces a novel approach to sampling from spike-and-slab priors by mapping the discrete spike into a continuous latent space, enabling the use of advanced dynamics-based samplers like Hamiltonian Monte Carlo for improved posterior inference.

Contribution

It proposes a new latent universe mapping for spike-and-slab priors, allowing the application of diverse dynamics-based samplers, including Hamiltonian Monte Carlo, to enhance sampling efficiency.

Findings

The latent universe mapping justifies replacing exponential sticking times with their expectation.

Hamiltonian sticky sampler can be derived as a limit of the original sticky process.

Empirical results show the new methods are at least as efficient as existing approaches.

Abstract

Combining a continuous "slab" density with discrete "spike" mass at zero, spike-and-slab priors provide important tools for inducing sparsity and carrying out variable selection in Bayesian models. However, the presence of discrete mass makes posterior inference challenging. "Sticky" extensions to piecewise-deterministic Markov process samplers have shown promising performance, where sampling from the spike is achieved by the process sticking there for an exponentially distributed duration. As it turns out, the sampler remains valid when the exponential sticking time is replaced with its expectation. We justify this by mapping the spike to a continuous density over a latent universe, allowing the sampler to be reinterpreted as traversing this universe while being stuck in the original space. This perspective opens up an array of possibilities to carry out posterior computation under spike-and-slab type priors. Notably, it enables us to construct sticky samplers using other dynamics-based paradigms such as Hamiltonian Monte Carlo; in fact, original sticky process can be established as a partial position-momentum refreshment limit of our Hamiltonian sticky sampler. Our theoretical and empirical findings suggest these alternatives to be at least as efficient as the original sticky approach.