Confidence Intervals for Random Forests: The Jackknife and the Infinitesimal Jackknife

TL;DR

This paper develops efficient methods for estimating the variability and standard errors of predictions made by random forests, using improved jackknife and infinitesimal jackknife techniques that require fewer bootstrap replicates.

Contribution

It introduces improved variance estimation methods for random forests that reduce computational cost by requiring fewer bootstrap replicates, and analyzes their sampling distributions.

Findings

IJ estimator needs 1.7 times fewer bootstrap replicates than jackknife.

Proposed methods require only on the order of n bootstrap replicates.

Experiments demonstrate the accuracy and efficiency of the new estimators.

Abstract

We study the variability of predictions made by bagged learners and random forests, and show how to estimate standard errors for these methods. Our work builds on variance estimates for bagging proposed by Efron (1992, 2012) that are based on the jackknife and the infinitesimal jackknife (IJ). In practice, bagged predictors are computed using a finite number B of bootstrap replicates, and working with a large B can be computationally expensive. Direct applications of jackknife and IJ estimators to bagging require B on the order of n^{1.5} bootstrap replicates to converge, where n is the size of the training set. We propose improved versions that only require B on the order of n replicates. Moreover, we show that the IJ estimator requires 1.7 times less bootstrap replicates than the jackknife to achieve a given accuracy. Finally, we study the sampling distributions of the jackknife and IJ variance estimates themselves. We illustrate our findings with multiple experiments and simulation studies.