Nearly Consistent Finite Particle Estimates in Streaming Importance Sampling

TL;DR

This paper introduces a method for maintaining a finite, nearly consistent set of importance sampling particles in Bayesian inference by embedding densities in RKHS and sequentially projecting them, balancing bias and memory.

Contribution

It proposes a novel online approach using RKHS embeddings and MMD-based projections to achieve nearly consistent finite particle estimates in importance sampling.

Findings

Achieves comparable accuracy with fewer particles

Provides a tunable bias-memory tradeoff

Reduces computational complexity significantly

Abstract

In Bayesian inference, we seek to compute information about random variables such as moments or quantiles on the basis of {available data} and prior information. When the distribution of random variables is {intractable}, Monte Carlo (MC) sampling is usually required. {Importance sampling is a standard MC tool that approximates this unavailable distribution with a set of weighted samples.} This procedure is asymptotically consistent as the number of MC samples (particles) go to infinity. However, retaining infinitely many particles is intractable. Thus, we propose a way to only keep a \emph{finite representative subset} of particles and their augmented importance weights that is \emph{nearly consistent}. To do so in {an online manner}, we (1) embed the posterior density estimate in a reproducing kernel Hilbert space (RKHS) through its kernel mean embedding; and (2) sequentially project this RKHS element onto a lower-dimensional subspace in RKHS using the maximum mean discrepancy, an integral probability metric. Theoretically, we establish that this scheme results in a bias determined by a compression parameter, which yields a tunable tradeoff between consistency and memory. In experiments, we observe the compressed estimates achieve comparable performance to the dense ones with substantial reductions in representational complexity.