Multirate Stein Variational Gradient Descent for Efficient Bayesian Sampling

TL;DR

This paper introduces a multirate version of Stein variational gradient descent (SVGD) that updates different components at different rates, improving robustness and efficiency in complex Bayesian inference tasks.

Contribution

The authors develop a multirate SVGD framework with practical algorithms, including adaptive and fixed variants, tailored for high-dimensional and complex posteriors.

Findings

Multirate SVGD improves robustness across diverse Bayesian inference problems.

Adaptive multirate SVGD performs best on stiff, anisotropic, and multimodal targets.

Fixed multirate SVGD offers a simpler, cost-effective alternative with strong performance.

Abstract

Many particle-based Bayesian inference methods use a single global step size for all parts of the update. In Stein variational gradient descent (SVGD), however, each update combines two qualitatively different effects: attraction toward high-posterior regions and repulsion that preserves particle diversity. These effects can evolve at different rates, especially in high-dimensional, anisotropic, or hierarchical posteriors, so one step size can be unstable in some regions and inefficient in others. We derive a multirate version of SVGD that updates these components on different time scales. The framework yields practical algorithms, including a symmetric split method, a fixed multirate method (MR-SVGD), and an adaptive multirate method (Adapt-MR-SVGD) with local error control. We evaluate the methods in a broad and rigorous benchmark suite covering six problem families: a 50D Gaussian target, multiple 2D synthetic targets, UCI Bayesian logistic regression, multimodal Gaussian mixtures, Bayesian neural networks, and large-scale hierarchical logistic regression. Evaluation includes posterior-matching metrics, predictive performance, calibration quality, mixing, and explicit computational cost accounting. Across these six benchmark families, multirate SVGD variants improve robustness and quality-cost tradeoffs relative to vanilla SVGD. The strongest gains appear on stiff hierarchical, strongly anisotropic, and multimodal targets, where adaptive multirate SVGD is usually the strongest variant and fixed multirate SVGD provides a simpler robust alternative at lower cost.