Boosting through Optimization of Margin Distributions

TL;DR

This paper introduces MDBoost, a new boosting algorithm that directly optimizes the margin distribution by maximizing average margins and minimizing variance, leading to improved generalization performance.

Contribution

The paper proposes a novel boosting method, MDBoost, which explicitly optimizes margin distribution, unlike traditional algorithms that focus on convex loss functions.

Findings

MDBoost outperforms AdaBoost and LPBoost on UCI datasets

It effectively maximizes average margin and minimizes margin variance

The algorithm uses a totally-corrective optimization approach based on column generation

Abstract

Boosting has attracted much research attention in the past decade. The success of boosting algorithms may be interpreted in terms of the margin theory. Recently it has been shown that generalization error of classifiers can be obtained by explicitly taking the margin distribution of the training data into account. Most of the current boosting algorithms in practice usually optimizes a convex loss function and do not make use of the margin distribution. In this work we design a new boosting algorithm, termed margin-distribution boosting (MDBoost), which directly maximizes the average margin and minimizes the margin variance simultaneously. This way the margin distribution is optimized. A totally-corrective optimization algorithm based on column generation is proposed to implement MDBoost. Experiments on UCI datasets show that MDBoost outperforms AdaBoost and LPBoost in most cases.