Narrowing the Gap: Random Forests In Theory and In Practice

TL;DR

This paper advances understanding of random forests by introducing a new theoretically tractable variant, proving its consistency, and empirically comparing it to practical models to analyze their differences.

Contribution

It presents a new consistent, theoretically tractable random regression forest model and compares it empirically with existing practical algorithms.

Findings

The new model is proven to be consistent.

Empirical results highlight differences between theoretical and practical models.

Insights into simplifications used in theoretical models.

Abstract

Despite widespread interest and practical use, the theoretical properties of random forests are still not well understood. In this paper we contribute to this understanding in two ways. We present a new theoretically tractable variant of random regression forests and prove that our algorithm is consistent. We also provide an empirical evaluation, comparing our algorithm and other theoretically tractable random forest models to the random forest algorithm used in practice. Our experiments provide insight into the relative importance of different simplifications that theoreticians have made to obtain tractable models for analysis.