Boosting Variational Inference With Locally Adaptive Step-Sizes

TL;DR

This paper introduces a novel approach to Boosting Variational Inference by utilizing locally adaptive step-sizes based on local curvature, improving efficiency and convergence over traditional methods.

Contribution

It proposes a new approximate backtracking algorithm to estimate local curvature, reducing resource requirements and enhancing convergence rates in Boosting Variational Inference.

Findings

The local curvature approach reduces memory and time consumption.

Theoretical convergence rates are improved with the new algorithm.

Experimental results validate the efficiency on synthetic and real datasets.

Abstract

Variational Inference makes a trade-off between the capacity of the variational family and the tractability of finding an approximate posterior distribution. Instead, Boosting Variational Inference allows practitioners to obtain increasingly good posterior approximations by spending more compute. The main obstacle to widespread adoption of Boosting Variational Inference is the amount of resources necessary to improve over a strong Variational Inference baseline. In our work, we trace this limitation back to the global curvature of the KL-divergence. We characterize how the global curvature impacts time and memory consumption, address the problem with the notion of local curvature, and provide a novel approximate backtracking algorithm for estimating local curvature. We give new theoretical convergence rates for our algorithms and provide experimental validation on synthetic and real-world datasets.