Boosting as Frank-Wolfe

TL;DR

This paper introduces a unified boosting framework based on the Frank-Wolfe algorithm, which generalizes existing boosting methods for soft margin optimization and improves practical performance by combining secondary algorithms.

Contribution

It presents a generic boosting scheme that unifies and extends LPBoost, ERLPBoost, and C-ERLPBoost using the Frank-Wolfe algorithm, allowing flexible incorporation of secondary algorithms.

Findings

The scheme retains convergence guarantees similar to existing methods.

Experiments show the scheme performs comparably to LPBoost in practice.

The framework offers a unified view of boosting algorithms for soft margin problems.

Abstract

Some boosting algorithms, such as LPBoost, ERLPBoost, and C-ERLPBoost, aim to solve the soft margin optimization problem with the $\ell_1$-norm regularization. LPBoost rapidly converges to an $\epsilon$-approximate solution in practice, but it is known to take $\Omega(m)$ iterations in the worst case, where $m$ is the sample size. On the other hand, ERLPBoost and C-ERLPBoost are guaranteed to converge to an $\epsilon$-approximate solution in $O(\frac{1}{\epsilon^2} \ln \frac{m}{\nu})$ iterations. However, the computation per iteration is very high compared to LPBoost. To address this issue, we propose a generic boosting scheme that combines the Frank-Wolfe algorithm and any secondary algorithm and switches one to the other iteratively. We show that the scheme retains the same convergence guarantee as ERLPBoost and C-ERLPBoost. One can incorporate any secondary algorithm to improve in practice. This scheme comes from a unified view of boosting algorithms for soft margin optimization. More specifically, we show that LPBoost, ERLPBoost, and C-ERLPBoost are instances of the Frank-Wolfe algorithm. In experiments on real datasets, one of the instances of our scheme exploits the better updates of the secondary algorithm and performs comparably with LPBoost.