The return of AdaBoost.MH: multi-class Hamming trees

TL;DR

This paper introduces a novel multi-class boosting method using vector-valued decision trees within AdaBoost.MH, avoiding reduction to binary problems, and demonstrates competitive performance with SVMs and existing boosting algorithms.

Contribution

It proposes a new multi-class boosting approach with vector-valued decision trees that maintains efficiency and effectiveness without problem reduction.

Findings

Performs on par with support vector machines.

Outperforms other AdaBoost.MH implementations.

Maintains computational efficiency of binary AdaBoost.

Abstract

Within the framework of AdaBoost.MH, we propose to train vector-valued decision trees to optimize the multi-class edge without reducing the multi-class problem to $K$ binary one-against-all classifications. The key element of the method is a vector-valued decision stump, factorized into an input-independent vector of length $K$ and label-independent scalar classifier. At inner tree nodes, the label-dependent vector is discarded and the binary classifier can be used for partitioning the input space into two regions. The algorithm retains the conceptual elegance, power, and computational efficiency of binary AdaBoost. In experiments it is on par with support vector machines and with the best existing multi-class boosting algorithm AOSOLogitBoost, and it is significantly better than other known implementations of AdaBoost.MH.