Nearly Optimal Bounds for Computing Decision Tree Splits in Data Streams

TL;DR

This paper presents nearly optimal one-pass algorithms and matching lower bounds for decision tree split approximation in data streams, improving space complexity for regression and classification tasks.

Contribution

It introduces new space-efficient algorithms with tight bounds for approximating decision tree splits in data streams, leveraging Lipschitz properties and sketching techniques.

Findings

Optimal space bounds for regression split approximation are established.

Improved space complexity for classification Gini split approximation is achieved.

Matching lower bounds confirm the tightness of the proposed algorithms.

Abstract

We establish nearly optimal upper and lower bounds for approximating decision tree splits in data streams. For regression with labels in the range $\{0,1,\ldots,M\}$, we give a one-pass algorithm using $\tilde{O}(M^2/\epsilon)$ space that outputs a split within additive $\epsilon$ error of the optimal split, improving upon the two-pass algorithm of Pham et al. (ISIT 2025). Furthermore, we provide a matching one-pass lower bound showing that $\Omega(M^2/\epsilon)$ space is indeed necessary. For classification, we also obtain a one-pass algorithm using $\tilde{O}(1/\epsilon)$ space for approximating the optimal Gini split, improving upon the previous $\tilde{O}(1/\epsilon^2)$-space algorithm. We complement these results with matching space lower bounds: $\Omega(1/\epsilon)$ for Gini impurity and $\Omega(1/\epsilon)$ for misclassification (which matches the upper bound obtained by sampling). Our algorithms exploit the Lipschitz property of the loss functions and use reservoir sampling along with Count--Min sketches with range queries. Our lower bounds follow from careful reductions from the INDEX problem.