Importance sampling for Bayesian inference: polynomial-dimension dependent error bounds

TL;DR

This paper analyzes importance sampling in high-dimensional Bayesian models, establishing conditions under which error bounds remain finite and converge at the standard Monte Carlo rate, regardless of dimension.

Contribution

It introduces a probabilistic framework using a link function to characterize error bounds, providing a new way to handle high-dimensional Bayesian inference.

Findings

Error bounds are finite and converge at O(N^{-1/2}) if the link function is Bochner integrable.

The approach offers a mechanism to control the dependence of error on model dimension.

Explicit error scaling examples are provided for linear-Gaussian and bounded observation models.

Abstract

Many Bayesian inference problems involve high-dimensional models where the performance of standard importance sampling (IS) methods often degrades rapidly as the dimensionality increases. Classical analyses of IS typically rely on the assumption that observations are arbitrary but fixed (i.e., deterministic), thereby neglecting the probabilistic structure that the Bayesian model induces on the data. In this paper, we adopt the perspective that observations are themselves random variables whose distribution is governed by the underlying model. Within this probabilistic framework, we identify a model-dependent function, referred to as the link function, which connects the fixed- and random-observation formulations. We provide a characterization of the $L^2$ Monte Carlo estimation error: specifically, we show that the $L^2$ error bounds are finite and converge at the standard Monte Carlo rate $O(N^{-1/2})$, for arbitrarily large dimension, if and only if the link function is Bochner integrable. This result reveals the fundamental quantity controlling the approximation error and establishes a mechanism to manage the dependence on the model state dimension. Consequently, our approach provides a principled way to alleviate the challenges of high dimensionality, offering insights that transcend worst-case analyses dominant in the existing literature. Finally, we derive explicit analytical examples of the dimensional scaling of the associated errors for several model classes, including linear-Gaussian systems and models with bounded observation functions.