On the Dual Formulation of Boosting Algorithms

TL;DR

This paper presents a dual formulation perspective of boosting algorithms, revealing their connection to entropy maximization and margin distribution, and introduces faster, totally corrective optimization algorithms.

Contribution

It uncovers the dual problems of boosting as entropy maximization and develops efficient, totally corrective algorithms with faster convergence.

Findings

Boosting algorithms' duals are entropy maximization problems.

AdaBoost approximately maximizes average margin.

Proposed algorithms converge faster with fewer weak classifiers.

Abstract

We study boosting algorithms from a new perspective. We show that the Lagrange dual problems of AdaBoost, LogitBoost and soft-margin LPBoost with generalized hinge loss are all entropy maximization problems. By looking at the dual problems of these boosting algorithms, we show that the success of boosting algorithms can be understood in terms of maintaining a better margin distribution by maximizing margins and at the same time controlling the margin variance.We also theoretically prove that, approximately, AdaBoost maximizes the average margin, instead of the minimum margin. The duality formulation also enables us to develop column generation based optimization algorithms, which are totally corrective. We show that they exhibit almost identical classification results to that of standard stage-wise additive boosting algorithms but with much faster convergence rates. Therefore fewer weak classifiers are needed to build the ensemble using our proposed optimization technique.