On the Equivalence between Herding and Conditional Gradient Algorithms

TL;DR

This paper reveals that herding is equivalent to a conditional gradient optimization method, allowing the use of convex optimization theory to analyze and improve herding for integral approximation and understanding its learning bias.

Contribution

It establishes the equivalence between herding and conditional gradient algorithms, enabling convergence analysis and development of faster variants.

Findings

Herding is mathematically equivalent to a conditional gradient algorithm.

Faster algorithms can outperform herding in integral approximation.

Herding often converges towards the maximum entropy distribution.

Abstract

We show that the herding procedure of Welling (2009) takes exactly the form of a standard convex optimization algorithm--namely a conditional gradient algorithm minimizing a quadratic moment discrepancy. This link enables us to invoke convergence results from convex optimization and to consider faster alternatives for the task of approximating integrals in a reproducing kernel Hilbert space. We study the behavior of the different variants through numerical simulations. The experiments indicate that while we can improve over herding on the task of approximating integrals, the original herding algorithm tends to approach more often the maximum entropy distribution, shedding more light on the learning bias behind herding.