Models under which random forests perform badly; consequences for applications

TL;DR

This paper identifies specific data models where random forests perform poorly or fail to converge, and proposes simple methods to improve their performance by focusing on variable importance.

Contribution

It introduces models causing random forest failures and suggests a practical approach to enhance performance using variable importance-based splits.

Findings

Random forests can be extremely slow or inconsistent on certain data models.

Simple importance-based splitting methods can significantly improve performance.

Numerical experiments support the effectiveness of the proposed approach.

Abstract

We give examples of data-generating models under which Breiman's random forest may be extremely slow to converge to the optimal predictor or even fail to be consistent. The evidence provided for these properties is based on mostly intuitive arguments, similar to those used earlier with simpler examples, and on numerical experiments. Although one can always choose models under which random forests perform very badly, we show that simple methods based on statistics of `variable use' and `variable importance' can often be used to construct a much better predictor based on a `many-armed' random forest obtained by forcing initial splits on variables which the default version of the algorithm tends to ignore.