Improving Random Forests by Smoothing

TL;DR

This paper introduces a kernel-based smoothing technique for random forests, enhancing their predictive smoothness and performance, especially with limited data, by combining tree adaptability with kernel regularity.

Contribution

It proposes a novel smoothing method that improves random forest predictions by integrating kernel smoothing, capturing uncertainty in tree cut points, and enhancing performance in small data regimes.

Findings

Smoothed random forests outperform standard ones in predictive accuracy.

The method effectively captures uncertainty in tree cut points.

Performance gains are especially notable in data-scarce scenarios.

Abstract

Random forest regression is a powerful non-parametric method that adapts to local data characteristics through data-driven partitioning, making it effective across diverse application domains. However, the piecewise constant nature of random forest predictions means each partition is predicted independently, ignoring potential smoothness in the underlying function. Particularly in the small data regime, this lack of information sharing across the input space can lead to suboptimal performance. In this work, we propose a kernel-based smoothing mechanism that enhances random forests by introducing local regularity to their predictions while preserving their adaptive partitioning capabilities. Our approach applies kernel smoothing to the piecewise constant outputs of random forests, effectively combining the adaptability of tree-based methods with the smoothness assumptions of kernel methods. We show that this smoothing procedure can be interpreted as capturing the variability/uncertainty in the tree cut points under resampling of the training inputs. Empirical results demonstrate that the proposed smoothed random forest model consistently improves predictive performance across diverse test cases, particularly in data-scarce settings. Code, datasets, and experiment results are publicly available at https://github.com/Neal-Liu-Ziyi/SmoothedRandomForest.git.