Coin Sampling: Gradient-Based Bayesian Inference without Learning Rates

TL;DR

This paper introduces new particle-based Bayesian inference methods based on coin betting that eliminate the need for learning rate tuning, achieving competitive results in high-dimensional models.

Contribution

It presents a novel class of learning-rate free particle-based inference algorithms using coin betting, improving scalability and ease of use.

Findings

Comparable performance to existing ParVI methods

Effective in high-dimensional models and datasets

No hyperparameter tuning required

Abstract

In recent years, particle-based variational inference (ParVI) methods such as Stein variational gradient descent (SVGD) have grown in popularity as scalable methods for Bayesian inference. Unfortunately, the properties of such methods invariably depend on hyperparameters such as the learning rate, which must be carefully tuned by the practitioner in order to ensure convergence to the target measure at a suitable rate. In this paper, we introduce a suite of new particle-based methods for scalable Bayesian inference based on coin betting, which are entirely learning-rate free. We illustrate the performance of our approach on a range of numerical examples, including several high-dimensional models and datasets, demonstrating comparable performance to other ParVI algorithms with no need to tune a learning rate.