To discretize continually: Mean shift interacting particle systems for Bayesian inference

TL;DR

This paper introduces a novel class of mean shift interacting particle systems for Bayesian inference that efficiently approximate expectations, handle multi-modality, and scale to high dimensions without requiring the normalizing constant.

Contribution

It extends classical mean shift algorithms to continuous distributions using MMD minimization, providing invariant, scalable, and versatile sampling methods.

Findings

Converges quickly and captures anisotropy and multi-modality.

Avoids mode collapse and scales to high dimensions.

Performs well on diverse benchmark problems.

Abstract

Integration against a probability distribution given its unnormalized density is a central task in Bayesian inference and other fields. We introduce new methods for approximating such expectations with a small set of weighted samples -- i.e., a quadrature rule -- constructed via an interacting particle system that minimizes maximum mean discrepancy (MMD) to the target distribution. These methods extend the classical mean shift algorithm, as well as recent algorithms for optimal quantization of empirical distributions, to the case of continuous distributions. Crucially, our approach creates dynamics for MMD minimization that are invariant to the unknown normalizing constant; they also admit both gradient-free and gradient-informed implementations. The resulting mean shift interacting particle systems converge quickly, capture anisotropy and multi-modality, avoid mode collapse, and scale to high dimensions. We demonstrate their performance on a wide range of benchmark sampling problems, including multi-modal mixtures, Bayesian hierarchical models, PDE-constrained inverse problems, and beyond.