Review and Demonstration of a Mixture Representation for Simulation from Densities Involving Sums of Powers

TL;DR

This paper reviews and demonstrates a latent variable representation for simulating from posterior distributions involving sums of powers, especially for cases where the power parameter is less than or equal to one, which are common in sparse and heavy-tailed regression models.

Contribution

It extends the latent variable simulation approach to cases where the power parameter is between 0 and 2, including the challenging case where q < 1, and demonstrates its application in Bayesian penalized regression.

Findings

The latent variable method is effective for q in (0, 2).

It enables simulation from posteriors with non-differentiable densities.

Application to Bayesian penalized regression shows practical utility.

Abstract

Penalized and robust regression, especially when approached from a Bayesian perspective, can involve the problem of simulating a random variable $\boldsymbol z$ from a posterior distribution that includes a term proportional to a sum of powers, $\|\boldsymbol z \|^q_q$, on the log scale. However, many popular gradient-based methods for Markov Chain Monte Carlo simulation from such posterior distributions use Hamiltonian Monte Carlo and accordingly require conditions on the differentiability of the unnormalized posterior distribution that do not hold when $q \leq 1$ (Plummer, 2023). This is limiting; the setting where $q \leq 1$ includes widely used sparsity inducing penalized regression models and heavy tailed robust regression models. In the special case where $q = 1$, a latent variable representation that facilitates simulation from such a posterior distribution is well known. However, the setting where $q < 1$ has not been treated as thoroughly. In this note, we review the availability of a latent variable representation described in Devroye (2009), show how it can be used to simulate from such posterior distributions when $0 < q < 2$, and demonstrate its utility in the context of estimating the parameters of a Bayesian penalized regression model.