Confidence Intervals for the Generalisation Error of Random Forests

TL;DR

This paper introduces improved confidence intervals for estimating the generalisation error of random forests using bootstrap-based methods, enhancing the reliability of out-of-bag error estimates without additional computational cost.

Contribution

It proposes novel confidence interval techniques based on the delta-method-after-bootstrap and jackknife-after-bootstrap for random forests, improving coverage accuracy.

Findings

Enhanced coverage properties over naive intervals

Effective in both real and simulated data

No need for additional trees to compute intervals

Abstract

Out-of-bag error is commonly used as an estimate of generalisation error in ensemble-based learning models such as random forests. We present confidence intervals for this quantity using the delta-method-after-bootstrap and the jackknife-after-bootstrap techniques. These methods do not require growing any additional trees. We show that these new confidence intervals have improved coverage properties over the naive confidence interval, in real and simulated examples.