On Variance Estimation of Random Forests with Infinite-Order U-statistics

TL;DR

This paper introduces a novel variance estimator for random forests based on a new Hoeffding decomposition perspective, ensuring unbiasedness, ratio consistency, and improved finite-sample performance for confidence intervals.

Contribution

It proposes a new unbiased variance estimator using a peak region dominance view, establishing ratio consistency and connecting with existing estimators.

Findings

The new estimator has lower bias in simulations.

It achieves targeted coverage rates for confidence intervals.

The method is theoretically justified with ratio consistency.

Abstract

Infinite-order U-statistics (IOUS) has been used extensively on subbagging ensemble learning algorithms such as random forests to quantify its uncertainty. While normality results of IOUS have been studied extensively, its variance estimation approaches and theoretical properties remain mostly unexplored. Existing approaches mainly utilize the leading term dominance property in the Hoeffding decomposition. However, such a view usually leads to biased estimation when the kernel size is large or the sample size is small. On the other hand, while several unbiased estimators exist in the literature, their relationships and theoretical properties, especially the ratio consistency, have never been studied. These limitations lead to unguaranteed performances of constructed confidence intervals. To bridge these gaps in the literature, we propose a new view of the Hoeffding decomposition for variance estimation that leads to an unbiased estimator. Instead of leading term dominance, our view utilizes the dominance of the peak region. Moreover, we establish the connection and equivalence of our estimator with several existing unbiased variance estimators. Theoretically, we are the first to establish the ratio consistency of such a variance estimator, which justifies the coverage rate of confidence intervals constructed from random forests. Numerically, we further propose a local smoothing procedure to improve the estimator's finite sample performance. Extensive simulation studies show that our estimators enjoy lower bias and archive targeted coverage rates.