Random Forest (RF) Kernel for Regression, Classification and Survival

TL;DR

This paper investigates the performance and properties of RF kernels derived from random forests across various tasks, demonstrating their competitiveness and potential for practical applications in regression, classification, and survival analysis.

Contribution

It provides a comprehensive evaluation of RF kernels' performance, comparing them with traditional kernels and RF, and explores their extensions and future research directions.

Findings

RF kernels are competitive with RF in high-dimensional noisy data scenarios.

RF kernels often outperform the Laplace kernel in simulations.

Real data examples illustrate practical benefits of RF kernels.

Abstract

Breiman's random forest (RF) can be interpreted as an implicit kernel generator,where the ensuing proximity matrix represents the data-driven RF kernel. Kernel perspective on the RF has been used to develop a principled framework for theoretical investigation of its statistical properties. However, practical utility of the links between kernels and the RF has not been widely explored and systematically evaluated.Focus of our work is investigation of the interplay between kernel methods and the RF. We elucidate the performance and properties of the data driven RF kernels used by regularized linear models in a comprehensive simulation study comprising of continuous, binary and survival targets. We show that for continuous and survival targets, the RF kernels are competitive to RF in higher dimensional scenarios with larger number of noisy features. For the binary target, the RF kernel and RF exhibit comparable performance. As the RF kernel asymptotically converges to the Laplace kernel, we included it in our evaluation. For most simulation setups, the RF and RFkernel outperformed the Laplace kernel. Nevertheless, in some cases the Laplace kernel was competitive, showing its potential value for applications. We also provide the results from real life data sets for the regression, classification and survival to illustrate how these insights may be leveraged in practice.Finally, we discuss further extensions of the RF kernels in the context of interpretable prototype and landmarking classification, regression and survival. We outline future line of research for kernels furnished by Bayesian counterparts of the RF.