Randomization Can Reduce Both Bias and Variance: A Case Study in Random Forests

TL;DR

This paper investigates how random forests can reduce both bias and variance, especially in high SNR settings, providing insights into their practical advantages over bagging and emphasizing the importance of tuning parameters.

Contribution

It demonstrates that random forests can reduce bias in addition to variance, especially in high SNR scenarios, and highlights the significance of tuning $mtry$ for optimal performance.

Findings

Random forests reduce bias and variance in high SNR settings.

Random forests outperform bagging when data patterns are captured.

Tuning $mtry$ is crucial for maximizing random forest effectiveness.

Abstract

We study the often overlooked phenomenon, first noted in \cite{breiman2001random}, that random forests appear to reduce bias compared to bagging. Motivated by an interesting paper by \cite{mentch2020randomization}, where the authors explain the success of random forests in low signal-to-noise ratio (SNR) settings through regularization, we explore how random forests can capture patterns in the data that bagging ensembles fail to capture. We empirically demonstrate that in the presence of such patterns, random forests reduce bias along with variance and can increasingly outperform bagging ensembles when SNR is high. Our observations offer insights into the real-world success of random forests across a range of SNRs and enhance our understanding of the difference between random forests and bagging ensembles. Our investigations also yield practical insights into the importance of tuning $mtry$ in random forests.