Totally Corrective Multiclass Boosting with Binary Weak Learners

TL;DR

This paper introduces a new optimization framework for multiclass boosting that uses primal-dual techniques to create totally-corrective algorithms, achieving faster convergence and comparable accuracy to existing methods.

Contribution

It derives dual formulations for AdaBoost.MO and AdaBoost.ECC, enabling the design of totally-corrective multiclass boosting algorithms with improved convergence speed.

Findings

Faster convergence compared to stage-wise boosting methods.

Achieves comparable generalization performance to state-of-the-art algorithms.

Maximizes margins more aggressively through totally-corrective updates.

Abstract

In this work, we propose a new optimization framework for multiclass boosting learning. In the literature, AdaBoost.MO and AdaBoost.ECC are the two successful multiclass boosting algorithms, which can use binary weak learners. We explicitly derive these two algorithms' Lagrange dual problems based on their regularized loss functions. We show that the Lagrange dual formulations enable us to design totally-corrective multiclass algorithms by using the primal-dual optimization technique. Experiments on benchmark data sets suggest that our multiclass boosting can achieve a comparable generalization capability with state-of-the-art, but the convergence speed is much faster than stage-wise gradient descent boosting. In other words, the new totally corrective algorithms can maximize the margin more aggressively.