Iterative Weak Learnability and Multi-Class AdaBoost

TL;DR

This paper introduces a recursive ensemble algorithm for multi-class classification that improves weak learnability conditions, guarantees convergence to correct labels, and provides exponential decay of misclassification probability.

Contribution

It strengthens weak learnability conditions for multi-class AdaBoost, ensuring convergence and providing bounds on generalization error, with a simpler check than previous methods.

Findings

Final hypothesis converges to the correct label with probability 1.

Misclassification probability decreases exponentially with training.

Generalization error is bounded similarly to AdaBoost.

Abstract

We construct an efficient recursive ensemble algorithm for the multi-class classification problem, inspired by SAMME (Zhu, Zou, Rosset, and Hastie (2009)). We strengthen the weak learnability condition in Zhu, Zou, Rosset, and Hastie (2009) by requiring that the weak learnability condition holds for any subset of labels with at least two elements. This condition is simpler to check than many proposed alternatives (e.g., Mukherjee and Schapire (2013)). As SAMME, our algorithm is reduced to the Adaptive Boosting algorithm (Schapire and Freund (2012)) if the number of labels is two, and can be motivated as a functional version of the steepest descending method to find an optimal solution. In contrast to SAMME, our algorithm's final hypothesis converges to the correct label with probability 1. For any number of labels, the probability of misclassification vanishes exponentially as the training period increases. The sum of the training error and an additional term, that depends only on the sample size, bounds the generalization error of our algorithm as the Adaptive Boosting algorithm.