The $f$-Divergence Expectation Iteration Scheme

TL;DR

This paper presents the $f$-EI$()$ algorithm, an iterative method for $f$-divergence minimization in Bayesian measures, with proven convergence and practical gradient-based computation in specific cases, enhancing variational techniques.

Contribution

The paper introduces the $f$-EI$()$ algorithm, a new iterative approach for $f$-divergence minimization with theoretical guarantees and simplified computations for certain divergences.

Findings

Algorithm guarantees systematic $f$-divergence decrease.

Convergence to an optimum is proven.

Empirical results show effectiveness in variational methods.

Abstract

This paper introduces the $f$-EI$(\phi)$ algorithm, a novel iterative algorithm which operates on measures and performs $f$-divergence minimisation in a Bayesian framework. We prove that for a rich family of values of $(f,\phi)$ this algorithm leads at each step to a systematic decrease in the $f$-divergence and show that we achieve an optimum. In the particular case where we consider a weighted sum of Dirac measures and the $\alpha$-divergence, we obtain that the calculations involved in the $f$-EI$(\phi)$ algorithm simplify to gradient-based computations. Empirical results support the claim that the $f$-EI$(\phi)$ algorithm serves as a powerful tool to assist Variational methods.