(Decision and regression) tree ensemble based kernels for regression and classification

TL;DR

This paper investigates the performance and properties of tree ensemble based kernels derived from random forests and gradient boosted trees, demonstrating their competitiveness and practical utility in regression and classification tasks.

Contribution

It systematically evaluates the interplay between tree ensemble kernels and their ensembles, highlighting their effectiveness especially in high-dimensional noisy data scenarios.

Findings

RF/GBT kernels perform well with continuous targets in high dimensions.

RF/GBT kernels are comparable to their ensembles for binary classification.

Tree ensemble kernels are a valuable addition to machine learning tools.

Abstract

Tree based ensembles such as Breiman's random forest (RF) and Gradient Boosted Trees (GBT) can be interpreted as implicit kernel generators, where the ensuing proximity matrix represents the data-driven tree ensemble kernel. Kernel perspective on the RF has been used to develop a principled framework for theoretical investigation of its statistical properties. Recently, it has been shown that the kernel interpretation is germane to other tree-based ensembles e.g. GBTs. However, practical utility of the links between kernels and the tree ensembles has not been widely explored and systematically evaluated. Focus of our work is investigation of the interplay between kernel methods and the tree based ensembles including the RF and GBT. We elucidate the performance and properties of the RF and GBT based kernels in a comprehensive simulation study comprising of continuous and binary targets. We show that for continuous targets, the RF/GBT kernels are competitive to their respective ensembles in higher dimensional scenarios, particularly in cases with larger number of noisy features. For the binary target, the RF/GBT kernels and their respective ensembles exhibit comparable performance. We provide the results from real life data sets for regression and classification to show how these insights may be leveraged in practice. Overall, our results support the tree ensemble based kernels as a valuable addition to the practitioner's toolbox. Finally, we discuss extensions of the tree ensemble based kernels for survival targets, interpretable prototype and landmarking classification and regression. We outline future line of research for kernels furnished by Bayesian counterparts of the frequentist tree ensembles.