A Powerful Random Forest Featuring Linear Extensions (RaFFLE)

TL;DR

RaFFLE is a novel random forest framework that incorporates PILOT trees with linear extensions, enhancing regression accuracy especially for linear relationships while maintaining computational efficiency.

Contribution

Introduces RaFFLE, integrating PILOT trees into random forests, with theoretical guarantees and improved empirical performance over existing methods.

Findings

RaFFLE outperforms CART, Lasso, Ridge, and XGBoost on diverse datasets.

Theoretical guarantees for consistency and faster convergence under linear data models.

Effective balance of accuracy and efficiency across linear and nonlinear regression tasks.

Abstract

Random forests are widely used in regression. However, the decision trees used as base learners are poor approximators of linear relationships. To address this limitation we propose RaFFLE (Random Forest Featuring Linear Extensions), a novel framework that integrates the recently developed PILOT trees (Piecewise Linear Organic Trees) as base learners within a random forest ensemble. PILOT trees combine the computational efficiency of traditional decision trees with the flexibility of linear model trees. To ensure sufficient diversity of the individual trees, we introduce an adjustable regularization parameter and use node-level feature sampling. These modifications improve the accuracy of the forest. We establish theoretical guarantees for the consistency of RaFFLE under weak conditions, and its faster convergence when the data are generated by a linear model. Empirical evaluations on 136 regression datasets demonstrate that RaFFLE outperforms the classical CART and random forest methods, the regularized linear methods Lasso and Ridge, and the state-of-the-art XGBoost algorithm, across both linear and nonlinear datasets. By balancing predictive accuracy and computational efficiency, RaFFLE proves to be a versatile tool for tackling a wide variety of regression problems.