Forest Garrote

TL;DR

This paper introduces a convex penalty method for tree ensembles, especially Random Forests, enabling variable selection and interpretability without sacrificing predictive accuracy, and requiring no additional tuning after model fitting.

Contribution

A novel Garrote-style convex penalty for tree ensembles that facilitates variable selection and interpretability while maintaining predictive performance without extra tuning.

Findings

Maintains predictive accuracy comparable to original Random Forests.

Enables visualization and interpretation of selected variables.

No additional tuning parameters needed after fitting.

Abstract

Variable selection for high-dimensional linear models has received a lot of attention lately, mostly in the context of l1-regularization. Part of the attraction is the variable selection effect: parsimonious models are obtained, which are very suitable for interpretation. In terms of predictive power, however, these regularized linear models are often slightly inferior to machine learning procedures like tree ensembles. Tree ensembles, on the other hand, lack usually a formal way of variable selection and are difficult to visualize. A Garrote-style convex penalty for trees ensembles, in particular Random Forests, is proposed. The penalty selects functional groups of nodes in the trees. These could be as simple as monotone functions of individual predictor variables. This yields a parsimonious function fit, which lends itself easily to visualization and interpretation. The predictive power is maintained at least at the same level as the original tree ensemble. A key feature of the method is that, once a tree ensemble is fitted, no further tuning parameter needs to be selected. The empirical performance is demonstrated on a wide array of datasets.