Provably Efficient Posterior Sampling for Sparse Linear Regression via Measure Decomposition

TL;DR

This paper introduces a provably efficient algorithm for sampling from the posterior distribution in sparse linear regression by decomposing it into a mixture of simpler measures, enabling practical and statistically attractive sampling.

Contribution

It proposes a novel measure decomposition approach that reduces complex multimodal posteriors to tractable log-concave mixtures, with proven polynomial-time efficiency under mild conditions.

Findings

Algorithm is practical and efficient in polynomial time.

Numerical simulations show attractive statistical properties.

Method outperforms some state-of-the-art sampling techniques.

Abstract

We consider the problem of sampling from the posterior distribution of a $d$-dimensional coefficient vector $\boldsymbol{\theta}$, given linear observations $\boldsymbol{y} = \boldsymbol{X}\boldsymbol{\theta}+\boldsymbol{\varepsilon}$. In general, such posteriors are multimodal, and therefore challenging to sample from. This observation has prompted the exploration of various heuristics that aim at approximating the posterior distribution. In this paper, we study a different approach based on decomposing the posterior distribution into a log-concave mixture of simple product measures. This decomposition allows us to reduce sampling from a multimodal distribution of interest to sampling from a log-concave one, which is tractable and has been investigated in detail. We prove that, under mild conditions on the prior, for random designs, such measure decomposition is generally feasible when the number of samples per parameter $n/d$ exceeds a constant threshold. We thus obtain a provably efficient (polynomial time) sampling algorithm in a regime where this was previously not known. Numerical simulations confirm that the algorithm is practical, and reveal that it has attractive statistical properties compared to state-of-the-art methods.