Certified Dimension Reduction for Bayesian Updating with the Cross-Entropy Method

TL;DR

This paper introduces a dimension reduction technique for Bayesian inverse problems using the cross-entropy method, enabling efficient posterior estimation in high-dimensional spaces by identifying low-dimensional subspaces through spectral analysis.

Contribution

It proposes a novel approach combining cross-entropy minimization with spectral analysis for dimension reduction in Bayesian inverse problems, improving computational efficiency.

Findings

Effective in high-dimensional parameter spaces

Reduces computational cost significantly

Maintains accuracy of posterior estimates

Abstract

In inverse problems, the parameters of a model are estimated based on observations of the model response. The Bayesian approach is powerful for solving such problems; one formulates a prior distribution for the parameter state that is updated with the observations to compute the posterior parameter distribution. Solving for the posterior distribution can be challenging when, e.g., prior and posterior significantly differ from one another and/or the parameter space is high-dimensional. We use a sequence of importance sampling measures that arise by tempering the likelihood to approach inverse problems exhibiting a significant distance between prior and posterior. Each importance sampling measure is identified by cross-entropy minimization as proposed in the context of Bayesian inverse problems in Engel et al. (2021). To efficiently address problems with high-dimensional parameter spaces we set up the minimization procedure in a low-dimensional subspace of the original parameter space. The principal idea is to analyse the spectrum of the second-moment matrix of the gradient of the log-likelihood function to identify a suitable subspace. Following Zahm et al. (2021), an upper bound on the Kullback-Leibler-divergence between full-dimensional and subspace posterior is provided, which can be utilized to determine the effective dimension of the inverse problem corresponding to a prescribed approximation error bound. We suggest heuristic criteria for optimally selecting the number of model and model gradient evaluations in each iteration of the importance sampling sequence. We investigate the performance of this approach using examples from engineering mechanics set in various parameter space dimensions.