Expected information gain estimation via density approximations: Sample allocation and dimension reduction

TL;DR

This paper introduces transport-based schemes for estimating expected information gain in complex Bayesian models, optimizing sample allocation, and employing dimension reduction to improve efficiency and accuracy in high-dimensional settings.

Contribution

It develops novel transport-based methods for EIG estimation, analyzes optimal sample allocation, and proposes dimension reduction techniques to enhance high-dimensional EIG estimation accuracy.

Findings

Optimal sample allocation improves MSE convergence rate.

Gradient-based bounds enable effective dimension reduction.

Numerical experiments demonstrate improved EIG estimation in high-dimensional inverse problems.

Abstract

Computing expected information gain (EIG) from prior to posterior (equivalently, mutual information between candidate observations and model parameters or other quantities of interest) is a fundamental challenge in Bayesian optimal experimental design. We formulate flexible transport-based schemes for EIG estimation in general nonlinear/non-Gaussian settings, compatible with both standard and implicit Bayesian models. These schemes are representative of two-stage methods for estimating or bounding EIG using marginal and conditional density estimates. In this setting, we analyze the optimal allocation of samples between training (density estimation) and approximation of the outer prior expectation. We show that with this optimal sample allocation, the mean squared error (MSE) of the resulting EIG estimator converges more quickly than that of a standard nested Monte Carlo scheme. We then address the estimation of EIG in high dimensions, by deriving gradient-based upper bounds on the mutual information lost by projecting the parameters and/or observations to lower-dimensional subspaces. Minimizing these upper bounds yields projectors and hence low-dimensional EIG approximations that outperform approximations obtained via other linear dimension reduction schemes. Numerical experiments on a PDE-constrained Bayesian inverse problem also illustrate a favorable trade-off between dimension truncation and the modeling of non-Gaussianity, when estimating EIG from finite samples in high dimensions.