Cost free hyper-parameter selection/averaging for Bayesian inverse problems with vanilla and Rao-Blackwellized SMC Samplers

TL;DR

This paper introduces a method using Sequential Monte Carlo samplers to efficiently select and average over hyper-parameters in Bayesian inverse problems, enabling hyper-parameter estimation and sensitivity analysis at minimal additional computational cost.

Contribution

The work demonstrates that SMC samplers can naturally approximate marginal likelihoods for hyper-parameters, facilitating hyper-parameter selection and averaging without extra computational burden.

Findings

Enables hyper-parameter selection via Empirical Bayes.

Allows hyper-parameter averaging with hyper-priors.

Provides prior sensitivity analysis at negligible cost.

Abstract

In Bayesian inverse problems, one aims at characterizing the posterior distribution of a set of unknowns, given indirect measurements. For non-linear/non-Gaussian problems, analytic solutions are seldom available: Sequential Monte Carlo samplers offer a powerful tool for approximating complex posteriors, by constructing an auxiliary sequence of densities that smoothly reaches the posterior. Often the posterior depends on a scalar hyper-parameter. In this work, we show that properly designed Sequential Monte Carlo (SMC) samplers naturally provide an approximation of the marginal likelihood associated with this hyper-parameter for free, i.e. at a negligible additional computational cost. The proposed method proceeds by constructing the auxiliary sequence of distributions in such a way that each of them can be interpreted as a posterior distribution corresponding to a different value of the hyper-parameter. This can be exploited to perform selection of the hyper-parameter in Empirical Bayes approaches, as well as averaging across values of the hyper-parameter according to some hyper-prior distribution in Fully Bayesian approaches. For FB approaches, the proposed method has the further benefit of allowing prior sensitivity analysis at a negligible computational cost. In addition, the proposed method exploits particles at all the (relevant) iterations, thus alleviating one of the known limitations of SMC samplers, i.e. the fact that all samples at intermediate iterations are typically discarded. We show numerical results for two distinct cases where the hyper-parameter affects only the likelihood: a toy example, where an SMC sampler is used to approximate the full posterior distribution; and a brain imaging example, where a Rao-Blackwellized SMC sampler is used to approximate the posterior distribution of a subset of parameters in a conditionally linear Gaussian model.