Logarithmic Time Online Multiclass prediction

TL;DR

This paper introduces a novel online decision tree method for multiclass classification that achieves logarithmic training and testing time complexity in the number of classes, suitable for large-scale applications.

Contribution

It proposes a new tree construction approach with a unique objective function for dynamic data partitioning, and an online algorithm that improves test error efficiently.

Findings

Online algorithm outperforms traditional methods in test error reduction

Constructs trees with low label entropy leaves under certain conditions

Achieves logarithmic time complexity in large-k classification tasks

Abstract

We study the problem of multiclass classification with an extremely large number of classes (k), with the goal of obtaining train and test time complexity logarithmic in the number of classes. We develop top-down tree construction approaches for constructing logarithmic depth trees. On the theoretical front, we formulate a new objective function, which is optimized at each node of the tree and creates dynamic partitions of the data which are both pure (in terms of class labels) and balanced. We demonstrate that under favorable conditions, we can construct logarithmic depth trees that have leaves with low label entropy. However, the objective function at the nodes is challenging to optimize computationally. We address the empirical problem with a new online decision tree construction procedure. Experiments demonstrate that this online algorithm quickly achieves improvement in test error compared to more common logarithmic training time approaches, which makes it a plausible method in computationally constrained large-k applications.