Slow-Growing Trees

TL;DR

This paper introduces Slow-Growing Trees (SGT), a novel approach that uses a learning rate to produce deep trees matching Random Forest performance, and unifies Boosted Trees and Random Forests under a common framework.

Contribution

It presents SGT as a new method that employs slow learning to improve tree-based models and offers a unifying perspective on boosting and bagging techniques.

Findings

SGT matches Random Forest performance on regression tasks.

Unifies Boosted Trees and Random Forests under a common framework.

Demonstrates effectiveness of slow learning in tree ensemble methods.

Abstract

Random Forest's performance can be matched by a single slow-growing tree (SGT), which uses a learning rate to tame CART's greedy algorithm. SGT exploits the view that CART is an extreme case of an iterative weighted least square procedure. Moreover, a unifying view of Boosted Trees (BT) and Random Forests (RF) is presented. Greedy ML algorithms' outcomes can be improved using either "slow learning" or diversification. SGT applies the former to estimate a single deep tree, and Booging (bagging stochastic BT with a high learning rate) uses the latter with additive shallow trees. The performance of this tree ensemble quaternity (Booging, BT, SGT, RF) is assessed on simulated and real regression tasks.