Selective Cascade of Residual ExtraTrees

TL;DR

SCORE is a new tree ensemble method that combines residual learning, regularized regression, and boosting to enhance prediction accuracy and explainability, showing competitive results and robustness.

Contribution

Introduces SCORE, a novel ensemble method integrating residual ExtraTrees, regularized variable selection, and boosting, with a new variable importance measure.

Findings

SCORE achieves comparable or better prediction accuracy than existing methods.

The variable importance measure for SCORE is effective and comparable to benchmarks.

SCORE's performance remains stable across different hyper-parameter settings.

Abstract

We propose a novel tree-based ensemble method named Selective Cascade of Residual ExtraTrees (SCORE). SCORE draws inspiration from representation learning, incorporates regularized regression with variable selection features, and utilizes boosting to improve prediction and reduce generalization errors. We also develop a variable importance measure to increase the explainability of SCORE. Our computer experiments show that SCORE provides comparable or superior performance in prediction against ExtraTrees, random forest, gradient boosting machine, and neural networks; and the proposed variable importance measure for SCORE is comparable to studied benchmark methods. Finally, the predictive performance of SCORE remains stable across hyper-parameter values, suggesting potential robustness to hyperparameter specification.