Approximating splits for decision trees quickly in sparse data streams

TL;DR

This paper introduces a fast approximation algorithm for finding near-optimal splits in decision trees tailored for sparse data streams, significantly reducing computation time while maintaining high accuracy.

Contribution

The paper presents a novel algorithm that approximates optimal decision tree splits efficiently for sparse binary data streams, improving speed over existing methods.

Findings

Achieves $(1 + eta)$ approximation for information gain in amortized $O(eta^{-1}(1 + m ext{log} d) ext{log} ext{log} n)$ time.

Achieves $(1 + eta)$ approximation for Gini index in amortized $O(eta^{-1} + m ext{log} d)$ time.

Outperforms baseline methods in experiments, providing faster and nearly optimal splits.

Abstract

Decision trees are one of the most popular classifiers in the machine learning literature. While the most common decision tree learning algorithms treat data as a batch, numerous algorithms have been proposed to construct decision trees from a data stream. A standard training strategy involves augmenting the current tree by changing a leaf node into a split. Here we typically maintain counters in each leaf which allow us to determine the optimal split, and whether the split should be done. In this paper we focus on how to speed up the search for the optimal split when dealing with sparse binary features and a binary class. We focus on finding splits that have the approximately optimal information gain or Gini index. In both cases finding the optimal split can be done in $O(d)$ time, where $d$ is the number of features. We propose an algorithm that yields $(1 + \alpha)$ approximation when using conditional entropy in amortized $O(\alpha^{-1}(1 + m\log d) \log \log n)$ time, where $m$ is the number of 1s in a data point, and $n$ is the number of data points. Similarly, for Gini index, we achieve $(1 + \alpha)$ approximation in amortized $O(\alpha^{-1} + m \log d)$ time. Our approach is beneficial for sparse data where $m \ll d$. In our experiments we find almost-optimal splits efficiently, faster than the baseline, overperforming the theoretical approximation guarantees.