Margins, Shrinkage, and Boosting

TL;DR

This paper demonstrates that AdaBoost and similar algorithms can approximate maximum margin classifiers through appropriate step size scaling, providing theoretical guarantees for shrinkage and regularization effects in boosting methods.

Contribution

It offers theoretical analysis showing how step size choices in boosting algorithms influence margin maximization and regularization, extending guarantees to various loss functions.

Findings

Scaling step sizes with a small constant improves margin guarantees.

Shrinkage procedures are theoretically justified for exponential and logistic losses.

Regularized line searches enhance margin performance.

Abstract

This manuscript shows that AdaBoost and its immediate variants can produce approximate maximum margin classifiers simply by scaling step size choices with a fixed small constant. In this way, when the unscaled step size is an optimal choice, these results provide guarantees for Friedman's empirically successful "shrinkage" procedure for gradient boosting (Friedman, 2000). Guarantees are also provided for a variety of other step sizes, affirming the intuition that increasingly regularized line searches provide improved margin guarantees. The results hold for the exponential loss and similar losses, most notably the logistic loss.