Consistent Estimation of Residual Variance with Random Forest Out-Of-Bag Errors

TL;DR

This paper develops and proves the consistency of residual variance estimators based on out-of-bag errors in random forests, addressing a key gap in regression uncertainty quantification.

Contribution

It introduces a novel residual variance estimator for random forests and establishes its theoretical consistency under L2-regularity assumptions.

Findings

The estimator is consistent for residual variance in random forests.

Provides theoretical foundation for uncertainty quantification in random forest regression.

Addresses a previously underexplored aspect of residual variance estimation in machine learning.

Abstract

The issue of estimating residual variance in regression models has experienced relatively little attention in the machine learning community. However, the estimate is of primary interest in many practical applications, e.g. as a primary step towards the construction of prediction intervals. Here, we consider this issue for the random forest. Therein, the functional relationship between covariates and response variable is modeled by a weighted sum of the latter. The dependence structure is, however, involved in the weights that are constructed during the tree construction process making the model complex in mathematical analysis. Restricting to L2-consistent random forest models, we provide random forest based residual variance estimators and prove their consistency.

Equations

$$Y$$

$${\mathcal{D}}_n=\{({\mathbf{X}}_i^{\top },Y_i)\in [0,1{]}^p\times \mathbb{R}:i=1,\dots ,n\},$$

$$m_{M,n}(\mathbf{x};{\Theta }_1,\dots {\Theta }_M,{\mathcal{D}}_n)=\frac{1}{M}\sum _{j=1}^M{m}_n(\mathbf{x};{\Theta }_j,{\mathcal{D}}_n)$$

$$m_n(\mathbf{x})=m_n(\mathbf{x};{\mathcal{D}}_n)={\mathbb{E}}_{\Theta }[m_n(\mathbf{x};\Theta ,{\mathcal{D}}_n)].$$

$$L_n(j,z)$$

$$-\frac{1}{N_n(A_{\ell }^{(k)})}\sum _{i=1}^n(Y_i-{\bar{Y}}_{A_{\ell ,L}^{(k)}}\text{mathds}1\{X_{ji}<z\}-{\bar{Y}}_{A_{\ell ,R}^{(k)}}\text{mathds}1\{X_{ji}\ge z\}{)}^2\text{mathds}1\{{\mathbf{X}}_i\in A_{\ell }^{(k)}\}$$

$$\underset{n\to \infty }{\lim }\mathbb{E}[(m_n(\mathbf{X})-m(\mathbf{X}){)}^2]=0,$$

$$m_{M,n}^{OOB}(\mathbf{x};{\Theta }_1,\dots ,{\Theta }_M)⟶m_n^{OOB}(\mathbf{x}),{\mathbb{P}}_{\Theta }\text{ - a.s.}\text{ as }M\to \infty .$$

$$\underset{n\to \infty }{\lim }\mathbb{E}[(m_n^{OOB}({\mathbf{X}}_i)-m({\mathbf{X}}_i){)}^2]=0.$$

$${\widehat{ϵ}}_i=Y_i-m_n^{OOB}({\mathbf{X}}_i),$$

$${\hat{\sigma }}_{RF}^2=\frac{1}{n}\sum _{i=1}^n({\widehat{ϵ}}_i-{\overline{ϵ}}_{\cdot }{)}^2$$

$${\hat{\sigma }}_{RF}^2\stackrel{L_1}{⟶}{\sigma }^2\text{as }n\to \infty .$$

$${\hat{\sigma }}_{RFboot}^2$$

$$=:{\hat{\sigma }}_{RF}^2-{\hat{R}}_B(m_n).$$

$${\hat{\sigma }}_{RFboot}^2\stackrel{L_1}{⟶}{\sigma }^2,\text{ as }n\to \infty .$$

$${\hat{R}}_B(m_n)\ge \frac{{\hat{\sigma }}_{RF}^2}{a_n^2}.$$

$${\hat{\sigma }}_{RFfast}^2={\hat{\sigma }}_{RF}^2\left(1-\frac{1}{a_n^2}\right)$$

$$Z_i\sim Bin(M,p_n)\text{ where }{p}_n=\{\begin{array}{ll}1-a_n/n& \text{ for subsampling}\\ (1-1/n{)}^n& \text{ for bootstrapping with replacement.}\end{array}$$

$$\frac{1}{Z_i(M)}\sum _{l=1}^{Z_i(M)}m(\mathbf{x};{\Theta }_l,{\mathcal{D}}_{n-1})\stackrel{a.s.}{⟶}{\mathbb{E}}_{\Theta }[m(\mathbf{x};\Theta ,{\mathcal{D}}_{n-1})]=:m_n^{OOB}(\mathbf{x}),M\to \infty .$$

$$\mathbb{E}[\frac{1}{n}\sum _{i=1}^n{\widehat{ϵ}}_i^2]$$

$$=\mathbb{E}[(Y_1-m({\mathbf{X}}_1){)}^2]+\frac{1}{n}\sum _{i=1}^n\{2\mathbb{E}[(Y_i-m({\mathbf{X}}_i))(m({\mathbf{X}}_i)-m_n^{OOB}({\mathbf{X}}_i))]+$$

$$\mathbb{E}[(m({\mathbf{X}}_i)-m_n^{OOB}({\mathbf{X}}_i){)}^2]\}$$

$$={\sigma }^2+\frac{1}{n}\sum _{i=1}^n\{2(\mathbb{E}[m({\mathbf{X}}_i)\mathbb{E}[Y_i|{\mathbf{X}}_i]]-\mathbb{E}[Y_i{m}_n^{OOB}({\mathbf{X}}_i)]-\mathbb{E}[m({\mathbf{X}}_i{)}^2]+$$

$$\mathbb{E}[m({\mathbf{X}}_i)\mathbb{E}[m_n^{OOB}({\mathbf{X}}_i)|{\mathbf{X}}_i]])+\mathbb{E}[(m({\mathbf{X}}_i)-m_n^{OOB}({\mathbf{X}}_i){)}^2]\}$$

$$={\sigma }^2+\frac{1}{n}\sum _{i=1}^n\{2(\mathbb{E}[m({\mathbf{X}}_i)\mathbb{E}[Y_i|{\mathbf{X}}_i]]-\mathbb{E}[m({\mathbf{X}}_i)m_n^{OOB}({\mathbf{X}}_i)]-\mathbb{E}[{ϵ}_i{m}_n^{OOB}({\mathbf{X}}_i)]-$$

$$\mathbb{E}[m({\mathbf{X}}_i{)}^2]+\mathbb{E}[m({\mathbf{X}}_i)\mathbb{E}[m_n^{OOB}({\mathbf{X}}_i)|{\mathbf{X}}_i]])+\mathbb{E}[(m({\mathbf{X}}_i)-m_n^{OOB}({\mathbf{X}}_i){)}^2]\}$$

$$={\sigma }^2+\frac{1}{n}\sum _{i=1}^n\{2(\mathbb{E}[m({\mathbf{X}}_i{)}^2]-\mathbb{E}[m({\mathbf{X}}_i)\mathbb{E}[m_n^{OOB}({\mathbf{X}}_i)|{\mathbf{X}}_i]]-\mathbb{E}[m({\mathbf{X}}_i{)}^2]+$$

$$\mathbb{E}[m({\mathbf{X}}_i)\mathbb{E}[m_n^{OOB}({\mathbf{X}}_i)|{\mathbf{X}}_i]])+\mathbb{E}[(m({\mathbf{X}}_i)-m_n^{OOB}({\mathbf{X}}_i){)}^2]\}$$

$$={\sigma }^2+\frac{1}{n}\sum _{i=1}^n\mathbb{E}[(m({\mathbf{X}}_i)-m_n^{OOB}({\mathbf{X}}_i){)}^2]$$

$$={\sigma }^2+\mathbb{E}[(m({\mathbf{X}}_1)-m_n^{OOB}({\mathbf{X}}_1){)}^2]⟶{\sigma }^2$$

$$0\le \mathbb{E}\left[{\left(\frac{1}{n}\sum _{i=1}^n{\widehat{ϵ}}_i\right)}^2\right]$$

$$=\frac{1}{n^2}\mathbb{E}[\sum _{i=1}^n({\Delta }_n({\mathbf{X}}_i)+{ϵ}_i{)}^2]+\frac{1}{n^2}\mathbb{E}[\sum _{i\ne j}({\Delta }_n({\mathbf{X}}_i)+{ϵ}_i)({\Delta }_n({\mathbf{X}}_j)+{ϵ}_j)]$$

$$=\frac{1}{n^2}\sum _{i=1}^n\mathbb{E}[{\Delta }_n({\mathbf{X}}_i{)}^2+2{ϵ}_i{\Delta }_n({\mathbf{X}}_i)+{ϵ}_i^2]+$$

$$+\frac{1}{n^2}\sum _{i\ne j}\mathbb{E}[{\Delta }_n({\mathbf{X}}_i){\Delta }_n({\mathbf{X}}_j)+{\Delta }_n({\mathbf{X}}_i){ϵ}_j+{\Delta }_n({\mathbf{X}}_j){ϵ}_i]$$

$$\le \frac{{\sigma }^2}{n}+\frac{1}{n^2}\sum _{i=1}^n\mathbb{E}[{\Delta }_n({\mathbf{X}}_i{)}^2]+\frac{1}{n^2}\sum _{i\ne j}\sqrt{\mathbb{E}[{\Delta }_n({\mathbf{X}}_i{)}^2]\mathbb{E}[{\Delta }_n({\mathbf{X}}_j{)}^2]}+$$

$$+\sigma \sqrt{\mathbb{E}[{\Delta }_n({\mathbf{X}}_i{)}^2]}+\sigma \sqrt{\mathbb{E}[{\Delta }_n({\mathbf{X}}_j{)}^2]}$$

$$\stackrel{id}{=}\frac{{\sigma }^2}{n}+\frac{\mathbb{E}[{\Delta }_n({\mathbf{X}}_1{)}^2]}{n}+(1-\frac{1}{n})(\mathbb{E}[{\Delta }_n({\mathbf{X}}_1{)}^2]+2\sigma \sqrt{\mathbb{E}[{\Delta }_n({\mathbf{X}}_1{)}^2]})$$

$$⟶0$$

$$m_n^{OOB}({\mathbf{X}}_i)$$

$$W_{n,j}^{OOB}({\mathbf{X}}_i)={\mathbb{E}}_{\Theta }\left[\frac{\text{mathds}1\{{\mathbf{X}}_j\in A_n({\mathbf{X}}_i;\Theta )\}}{N_n(A_n({\mathbf{X}}_i;\Theta ))}\right],$$

Full text

††journal: Journal of LaTeX Templates

1 Introduction

Random forest models are non-parametric regression resp. classification trees that highly rely on the idea of bagging and feature sub-spacing during tree construction. This way, one aims to construct highly predictive models by averaging (for continuous outcomes) or taking majority votes (for categorical outcomes) over CART trees constructed on bootstrapped samples. At each node of a tree, the best cut is selected by optimizing a CART-split criterion such as the Gini impurity (for classification) or the squared prediction error (for regression) over a subsample of the feature space. This methodology has been proven to work well in predicting new outcomes as first shown in [1]. Despite that and closely related to the prediction of a new instance is the question how reliable this prognosis is. For example in [2], random forest models have been used in predicting disease risk in highly imbalanced data. Beyond point estimators, however, little information was known about the dispersion of disease risk prediction. In fact, estimating residual variance based upon machine learning techniques has experienced less attention compared to the extensive investigations on pure prediction. One exception is given in [3], where bootstrap corrected residual variance estimators are proposed. Moreover, they are analyzed in a simulation study for regression problems but no theoretical guarantees such as consistency have been proven. A similar observation holds for the jackknife-type sampling variance estimators given in [4]. In the present paper we will close this gap by investigating the theoretical properties of a new residual variance estimator within the random forest framework. The estimator is inspired by the one proposed in [3] and is shown to be consistent for estimating residual variance if the random forest estimate for the regression function is $L_{2}$-consistent. Thereby, our theoretical derivations are build upon existing results in the literature.

First theoretical properties of the random forest method such as ($L_{2}$-) consistency have already been proven in [1] while connections to layered nearest neighbors were made in [5] and [6]. The early consistency results were later extended by several authors [7, 8, 4, 9, 10]; particularly allowing for stronger results (as central limit theorems) or a more reasonable mathematical model that better approximates the true random forest approach. In particular, varying mathematical forces such as feature sub-spacing, bagging and the tree construction process make the analysis of the true random forest as applied in practice very complicated.

In the current work, we therefore decided to build upon the mathematical description of the random forest method as described in [9]. This allows the applicability of our estimator for a wide range of functional relationships while also incorporating relevant features of the algorithm such as the split-criterion.

Our paper is structured as follows. In the next section, we give a brief overview of the random forest and state the model framework. In addition, consistency results are stated. In the third section, we provide a residual variance estimate and prove its consistency in $L_{1}$-sense. Furthermore, bias-corrected residual variance estimators are proposed. Note that all proofs can be found in the appendix.

2 Model Framework and Random Forest

Our framework is the $L_{2}$ regression estimation in which the covariable vector $\mathbf{X}$ is assumed to lie on the $p$-dimensional unit-cube, i.e. $\mathbf{X}\in[0,1]^{p}$. Of primary interest in the current paper is the estimation of the residual variance $\sigma^{2}$ in a functional relation of the form

[TABLE]

Here, $\mathbb{E}[|m(\mathbf{X})|^{2}]<\infty$, $\mathbb{E}[\epsilon]=0$ and $Var(\epsilon)\equiv\sigma^{2}<\infty$ with $m:{\mathbb{R}}^{p}\rightarrow{\mathbb{R}}$ and $\epsilon$ is independent of $\mathbf{X}$. Given a training set

[TABLE]

of i.i.d. pairs $(\mathbf{X}_{i},Y_{i})$, $i=1,\dots,n$, we aim to deliver an estimate $\hat{\sigma}_{n}^{2}$ that is at least $L_{1}$-consistent. The construction of $\hat{\sigma}_{n}^{2}$ will be based on the random forest estimate $m_{n}:[0,1]^{p}\rightarrow{\mathbb{R}}$ approximating the regression function $m$. In the sequel, we will stick to the notation as given in [9] and shortly introduce the random forest model and corresponding mathematical forces involved in it.

The random forest model for regression is a collection of $M\in{\mathbb{N}}$ regression trees, where for each tree, a bootstrap sample is taken from $\mathcal{D}_{n}$ using with or without replacement procedures. This is denoted as the resampling strategy $\mathcal{S}$. Other sampling strategies than these two within the random forest model have been considered in [11], for example. Furthermore, at each node of the tree, feature sub-spacing is conducted selecting $m_{try}\in\{1,\dots,p\}$ features for possible split direction. Denote with $\Theta$ the generic random variable responsible for both, the bootstrap sample construction and the feature sub-spacing procedure. Then, $\Theta_{1},\dots,\Theta_{M}$ are assumed to be independent copies of $\Theta$ responsible for this random process in the $j$-th tree, independent of $\mathcal{D}_{n}$. The combination of the trees is conducted through averaging. i.e.

[TABLE]

and is referred to as the finite forest estimate of $m$. As explained in [9], the strong law of large numbers (for $M\rightarrow\infty$) allows to study $\mathbb{E}_{\Theta}[m_{n}(\mathbf{x};\Theta,\mathcal{D}_{n})]$ instead of $(\ref{FiniteForest})$. Hence, we set

[TABLE]

Similar to [9], we refer to the random forest algorithm by identfiying three parameters responsibly for the random forest tree construction:

• •

$m_{try}\in\{1,\dots,p\}$ the number of pre-selected directions for splitting,

• •

$a_{n}\in\{1,\dots,n\}$, the number of sampled points in the bootstrap step and

• •

$t_{n}\in\{1,\dots,a_{n}\}$, the number of leaves in each tree.

Let $\{A_{\ell}^{(k)}\}_{\ell=1}^{2^{k-1}}$ be a sequence of generic cells in ${\mathbb{R}}^{p}$ obtained at tree depth $k\in{\mathbb{N}}$, $k\leq\lceil\log_{2}(t_{n})\rceil+1$ and denote by $N_{n}(A_{\ell}^{(k)})$ the number of observations falling in $A_{\ell}^{(k)}$, where we set $A_{1}^{(1)}=[0,1]^{p}$. Here, we denote a cut as the pair $(j,z)\in\{1,\dots,p\}\times[0,1]$, where $j$ represents the selected variable in which its domain is cut at $z$. Furthermore, let $\mathcal{C}_{A_{\ell}^{(k)}}$ be the set of all possible cuts in $A_{\ell}^{(k)}$. It should be noted that the restriction of the feature domain to the $p$-dimensional unit-cube $[0,1]^{p}$ is no restriction since the random forest is invariant under monotone transformations.

Then formally, the random forest algorithm constructs decision trees resulting in regression estimators according to the following algorithm:

In order to establish $L_{1}$-consistency of the residual variance estimate $\hat{\sigma}_{n}^{2}$, we require at least $L_{2}$-consistency of the random forest method. That is,

[TABLE]

where the expectation is taken with respect to $\mathbf{X}$ and $\mathcal{D}_{n}$. Here, $(\mathbf{X},Y)$ is an independent copy of $(\mathbf{X}_{i},Y_{i})$ for $i\in\{1,\dots,n\}$.

Several authors attempted to prove that (4) is valid, i.e. that random forests are consistent in $L_{2}$-sense. [8], for example, assumed a simplified version of the random forest assuming that cuts happen independent of the response variable $Y$ in a purely random fashion. [9], established consistency of the original random forest by assuming that $m$ is the additive expansion of continuous functions on the unit cube. Therein, proofs have been provided for fully grown trees ($t_{n}=a_{n}$) and not fully grown trees $(t_{n}<a_{n})$ making additional assumptions on the asymptotic relation between $t_{n}$ and $a_{n}$. For example, Theorem 1 in [9] guarantees condition (4) for additive Gaussian regression models provided that $\mathbf{X}_{i}\stackrel{{\scriptstyle iid}}{{\sim}}Unif([0,1]^{p})$ and $a_{n}\rightarrow\infty$, $t_{n}\rightarrow\infty$, $t_{n}(\log(a_{n}))^{9}/a_{n}\rightarrow 0$ such that the resampling strategy $\mathcal{S}$ is restricted to sampling without replacement. In this context it should be noted that assumption (4) does not automatically lead to pointwise consistency, since the latter is rather hard to prove for random forest models and counterexamples exist on the original random forest model as mentioned in [12].

Anyhow, predicting outcomes among the training set $\mathcal{D}_{n}$ using the random forest is usually done by using Out-Of-Bag (OOB) subsamples. That is, averaging does not happen over all $M$ trees but over those trees that did not have the corresponding data point in their resampled data set during tree construction. This way, one aims to deliver unbiased estimators for predicted values. In addition, OOB samples have the advantage of delivering internal accuracy estimates, without separating the sample $\mathcal{D}_{n}$ into a training and test set. This way, the training sample size can be left sufficiently large. From a mathematical perspective, OOB-estimators of random forest have the nice property that independence between observed responses $Y_{i}$ and predicted $\hat{Y}_{i}$ remains valid for $i\in\{1,\dots,n\}$. This, because the prediction $\hat{Y}_{i}$ of $Y_{i}$ is based on samples not containing the point $(\mathbf{X}_{i}^{\top},Y_{i})$ for fixed $i=1,\dots,n$. Thus, the independence property directly results from the independence assumption given in (2). However, the justification to analyze infinite forests instead of finite forests as in (3) is unclear for OOB-estimates, since one does not consider the average over $M$ decision trees, but rather a random subset of $\{1,\dots,M\}$, depending on the data point one aims to predict. If we denote with $\hat{Y}_{i}=m_{n}^{OOB}(\mathbf{X}_{i})$ the OOB prediction of $\mathbf{X}_{i}$, for $i\in\{1,\dots,n\}$ and $m_{M,n}^{OOB}(\mathbf{X}_{i};\Theta_{1},\dots,\Theta_{M})$ the corresponding finite forest estimate, then we provide our first result proving the justification of considering infinite forests even for OOB samples.

Lemma 1**.**

Under Model (1), OOB predictions of finite forests are consistent, that is for all $\mathbf{x}\in[0,1]^{p}$

[TABLE]

The consistency assumption in (4) implies the consistency of the corresponding OOB-estimate. That is:

Corollary 1**.**

For every fixed $i\in\{1,\dots,n\}$ under Model (1) and assuming (4), OOB-estimators $m_{n}^{OOB}$ based on random forests are $L_{2}$-consistent in the following sense

[TABLE]

These preliminary results allow the construction of a consistent residual variance estimator based on OOB samples.

3 Residual Variance Estimation

We estimate the residuals based on OOB samples, i.e. we set for $i=1,\dots,n$

[TABLE]

which we denote as OOB-estimated residuals. Their sample variance

[TABLE]

or OOB-estimated residual variance is our proposed estimator. Here, $\bar{\epsilon}_{\cdot}=\sum_{i=1}^{n}\hat{\epsilon}_{i}/n$ denotes the mean of $\{\hat{\epsilon}_{i}\}_{i=1}^{n}$. A similar estimator has been proposed in [3], for which simulation studies on some functional relationships between $\mathbf{X}$ and $Y$ were considered for practical implementation. The next result guarantees asymptotic unbiasedness and consistency of $\hat{\sigma}_{RF}^{2}$ under Assumption (4).

Theorem 1**.**

Assume regression Model (1) and that (4) is valid. Then the residual variance estimate $\hat{\sigma}_{RF}^{2}$ given in (7) is asymptotically unbiased and $L_{1}$-consistent as $n\to\infty$, i.e.

[TABLE]

Remark 1** (Key Assumptions and Other Machine Learning Techniques).**

**

*(a) Beyond Assumption (4) the structure of the random forest is only used to prove (5) and to maintain that the error variables $\epsilon_{i}$ are independent from $m_{n}^{OOB}(\mathbf{X}_{i})$ and to have $m_{n}^{OOB}(\mathbf{X}_{i})\stackrel{{\scriptstyle d}}{{=}}m_{n}^{OOB}(\mathbf{X}_{j})$, both for all fixed $1\leq i,j\leq n$. Thus, the results can be extended to all methods guaranteeing these assumptions.

(b) Moreover, carefully checking the proof of Theorem 1, the independence of $\epsilon$ towards $\mathbf{X}$ can also be substituted by $\mathbb{E}[\epsilon|\mathbf{X}]=0$, $Var(\epsilon|\mathbf{X})\equiv\sigma^{2}$ while still maintaining the consistency result.

3.1 Bias-corrected Estimation

As explained in [3], the estimator (7) may be biased for finite sample size $n$. To this end, [3] proposed a biased-corrected version of $\hat{\sigma}_{RF}^{2}$ via parametric bootstrapping. Their idea is as follows: Given the data $\mathcal{D}_{n}$ generate i.i.d. parametric bootstrap residuals $\epsilon_{i,b}^{*},i=1,\dots,n,b=1,\dots,B,$ independent from $\Theta,\dots,\Theta_{M}$, $\mathcal{D}_{n}$, with mean $\mathbb{E}^{*}[\epsilon_{1,1}^{*}]=\mathbb{E}[\epsilon_{1,1}^{*}|\mathcal{D}_{n}]=0$ and variance $Var^{*}(\epsilon_{1,1}^{*})=Var(\epsilon_{1,1}^{*}|\mathcal{D}_{n})=\hat{\sigma}_{RF}^{2}$ from a parametric distribution with finite second moment, e.g. the normal distribution. Then, a bias-corrected bootstrap version of $\hat{\sigma}_{RF}^{2}$ is given by

[TABLE]

Here, $m_{n,b}^{OOB}$ is the OOB-estimation of $m$ using the tree structure of $m_{n}^{OOB}$ and feeding it with the bootstrapped sample $\mathcal{D}_{n,b}^{*}:=\{(\mathbf{X}_{i},Y_{i,b}^{*}):i=1,\dots,n\}$ in which terminal node values are substituted with corresponding $Y_{i,b}^{*}$’s where $Y_{i,b}^{*}=m_{n}^{OOB}(\mathbf{X}_{i})+\epsilon_{i,b}^{*}$.

In the following, we provide two important results regarding the bias-corrected version of $\hat{\sigma}_{RF}^{2}$. In Theorem 2, we prove that the bias-corected estimator in (3.1) is $L_{1}$-consistent. This guarantees that the proposed bootstrapping scheme does not systematically inflate our estimate. However, $\hat{\sigma}_{RFboot}^{2}$ comes with additional computation costs. Therefore, in Theorem 3, we provide an asymptotic lower bound which enables a fast, bias-corrected estimation of $\sigma^{2}$ for finite sample sizes.

Theorem 2**.**

Consider the random forest based parametric bootstrapping scheme as described for the estimation of (3.1). Assume further that the resampling strategy $\mathcal{S}$ is restricted to sampling without replacement. Assuming Model (1) and condition (4) with $a_{n}^{2}/n\longrightarrow 0$, as $n\rightarrow\infty$. Then $\hat{\sigma}_{RFboot}^{2}$ is asymptotically $L_{1}$-consistent, that is

[TABLE]

Theorem 3**.**

Consider the parametric bootstrapping scheme as described for the estimate in (3.1). Then for the random forest model, the follwoing inequality holds almost surely conditional on $\mathcal{D}_{n}$ as $B\rightarrow\infty$

[TABLE]

The result in Theorem 3 leads to a residual variance estimate that is computationally cheaper than the corresponding bootstrapped version, i.e. one can consider

[TABLE]

instead of $\hat{\sigma}_{RFboot}^{2}$, while saving considerable memory and computational time costs. This will lead to $\hat{\sigma}_{RF}^{2}\geq\hat{\sigma}_{RFfast}^{2}\geq\hat{\sigma}_{RFboot}^{2}$ almost surely.

4 Conclusion

The random forest is known as a powerful tool in applied data analysis for classification, regression and variable selection [13, 14, 15, 16, 17, 2]. Beyond its practical use, corresponding theoretical properties have been investigated under various conditions [1, 8, 6, 12, 9] covering topics such as the $L_{2}$-consistent estimation of the regression function. However, a comprehensive treatment on how to estimate corresponding dispersion parameters as the variance is almost not to be found in the literature.

An exception is given by the residual variance estimators proposed and examined in simulations in [3]. In the present paper, we complement their analyses by theoretically investigating residual variance estimators in regression models. To this end, we first show that analyzing the infinite forest estimate is legitimate, even when switching to OOB samples. This allows us to prove consistency of the OOB-errors’ sample variance in the $L_{1}$-sense if the random forest regression function estimate is assumed to be $L_{2}$-consistent. In addition, we also give some theoretical insight on the bias corrected residual variance estimate for finite samples as proposed in [3].

As the structure of the random forest is only needed to maintain the independence property in OOB samples, the current approach is also valid for any method that provides $L_{2}$-consistent regression function estimates.

5 Appendix.

In this section we state the proofs for Lemma 1, Corollary 1, Theorem 1, Theorem 2 and Theorem 3.

Proof of Lemma 1.

Let $i\in\{1,\dots,n\}$ be fixed and $\mathbf{x}\in[0,1]^{p}$ be an arbitrary and fixed point in the unit cube. Consider $m_{n}(\mathbf{x})=\mathbb{E}_{\Theta}[m_{n}(\mathbf{x};\Theta,\mathcal{D}_{n})]$. If we denote with $Z_{i}=Z_{i}(M)$ the number of the $M$ regression trees not containing the $i$-th observation, then it follows

[TABLE]

such that $p_{n}>0$. Since $Z_{i}=\sum\limits_{\ell=1}^{M}B_{\ell}$, where $B_{\ell}\stackrel{{\scriptstyle iid}}{{\sim}}Bernoulli(p_{n})$, it follows by the strong law of large numbers for fixed $n$ that $Z_{i}/M\stackrel{{\scriptstyle a.s.}}{{\rightarrow}}\mathbb{E}[B_{1}]=p_{n}$ as $M\rightarrow\infty$. Hence $Z_{i}(M)\stackrel{{\scriptstyle a.s.}}{{\longrightarrow}}\infty$, as $M\rightarrow\infty$ for fixed $p_{n}>0$. For given $\mathcal{D}_{n}^{(i)}:=\mathcal{D}_{n}\setminus\{(\mathbf{X}_{i},Y_{i})\}$, this justifies the consideration of

[TABLE]

∎

Proof of Corollary 1.

Let $i\in\{1,\dots,n\}$ be fixed. Define the reduced sample $\mathcal{D}_{n}^{-(i)}:=\mathcal{D}_{n}\setminus\{(\mathbf{X}_{i}^{\top},Y_{i})\}$ as the OOB-sample of $i$. Then it follows from the independence assumption in (2), that $(\mathbf{X}_{i}^{\top},Y_{i})$ is independent of $\mathcal{D}_{n}^{-(i)}$. Hence, $(\mathbf{X}_{i}^{\top},Y_{i})$ can be treated as an independent copy of $\mathcal{D}_{n}^{-(i)}$. The result thus follows immediately from (4). ∎

Proof of Theorem 1.

Consider $\hat{\epsilon}_{i}=Y_{i}-m_{n}^{OOB}(\mathbf{X}_{i})$ and $\hat{\sigma}_{RF}^{2}=\frac{1}{n}\sum\limits_{i=1}^{n}(\hat{\epsilon}_{i}^{2}-\bar{\epsilon}_{\cdot})^{2}=\frac{1}{n}\sum\limits_{i=1}^{n}\hat{\epsilon}_{i}^{2}-\bar{\epsilon}_{\cdot}^{2}$ from (6)–(7). Using Corollary 1 and independence of $\epsilon_{i}$ and $m_{n}^{OOB}(\mathbf{X}_{i})$ for all $i=1,\dots,n$ it follows that

[TABLE]

by Corollary 1 as $n\to\infty$. The second and last equality follows from the identical distribution of the sequences $\{Y_{i}-m(\mathbf{X}_{i})\}_{i=1}^{n}$ resp. $\{m(\mathbf{X}_{i})-m_{n}^{OOB}(\mathbf{X}_{i})\}_{i=1}^{n}$.

Furthermore, let $\varDelta_{n}(\mathbf{X}_{i}):=m(\mathbf{X}_{i})-m_{n}^{OOB}(\mathbf{X}_{i})$. Then using the Cauchy-Schwarz inequality we obtain

[TABLE]

by Corollary 1 as $n\to\infty$ which completes the proof.

∎

Proof of Theorem 2.

To be mathematically precise, let $\Theta,\dots,\Theta_{M}$ and $(Y_{i},{\bf X}_{i})$ be defined on some probability space $(\Omega,\mathcal{A},\mathbb{P})$ and let the parametric bootstrap variables $\epsilon_{i,b}^{*}$ be defined on another probability space $(\Omega^{*},\mathcal{A}^{*},\mathbb{P}^{*})$. Then, all random variables can be defined (via projections) on the joined product space $(\Omega\times\Omega^{*},\mathcal{A}\otimes\mathcal{A}^{*},\mathbb{P}\otimes\mathbb{P}^{*})$; explaining the assumption that the random variables $\epsilon_{i,b}^{*}$ are independent from $\mathcal{D}_{n}$ and i.i.d. generated from a distribution with finite second moment with $Var^{*}(\epsilon_{1,1}^{*})=\hat{\sigma}_{RF}^{2}$ and $\mathbb{E}^{*}[\epsilon_{1,1}^{*}]=0$.

Within this framework consider $Y_{i,b}^{*}=m_{n}^{OOB}(\mathbf{X}_{i})+\epsilon_{i,b}^{*}$ and denote with $\mathcal{D}_{n,b}^{*}:=\{(\mathbf{X}_{i},Y_{i,b}^{*}):i=1,\dots,n\}$ the set of the $b$-th bootstrapped sample for $b\in\{1,\dots,B\}$. Then the sequence of sets $\{\mathcal{D}_{n,b}^{*}\}_{b=1}^{B}$ is independent. In particular, conditioned on $\mathcal{D}_{n}$, $\{Y_{i,b}^{*}\}_{b}$ forms a sequence of i.i.d. random variables.

Now, note that random forest models are the weighted sum of the response variable. Hence, denoting with $A_{n}({\mathbf{x};\Theta})\subset[0,1]^{p}$ the hyper-rectangle obtained after constructing one random decision tree with seed parameter $\Theta$ containing $\mathbf{x}$, then the infinite random forest model can be rewritten as

[TABLE]

see, e.g., the proof of Theorem 2 in [9] for a similar observation. Here, $\sum\limits_{j=1}^{n}W_{n,j}^{OOB}(\mathbf{X}_{i})=1$ holds almost surely and the weights $W_{n,j}^{OOB}$ are defined as

[TABLE]

where $N_{n}(A_{n}(\mathbf{X}_{i};\Theta))$ is the number of data points falling in $A_{n}(\mathbf{X}_{i};\Theta)$. Further let $\mathbf{X}_{j}\stackrel{{\scriptstyle\Theta}}{{\leftrightarrow}}\mathbf{X}_{i}$ be the event that both points, $\mathbf{X}_{i}$ and $\mathbf{X}_{j}$, fall in the same cell under the tree constructed by $\Theta$. Due to sampling without replacement, there are ${n-1}\choose{a_{n}-1}$ choices to pick a fixed observation $\mathbf{X}_{i}$. Therefore, we obtain

[TABLE]

Setting $A_{n,i}=\sum\limits_{j=1}^{n}W_{nj}^{OOB}(\mathbf{X}_{i})\cdot(m_{n}^{OOB}(\mathbf{X}_{j})-m(\mathbf{X}_{j}))$ and $B_{n,i}=\sum\limits_{j=1}^{n}W_{nj}^{OOB}(\mathbf{X}_{i})\cdot(\epsilon_{j}^{*}-\epsilon_{j})$, we obtain the following result for every fixed $b\in\{1,\dots,B\}$ using the Cauchy-Schwarz inequality:

[TABLE]

In order to prove $L_{1}$-consistency of the bootstrapped corrected estimate, based on (5), we only need to show that $\mathbb{E}[A_{n,i}^{2}]\rightarrow 0$ and $\mathbb{E}[B_{n,i}^{2}]\rightarrow 0$ as $n\rightarrow\infty$. Now, note that $\mathbb{E}[(\epsilon_{j}^{*}-\epsilon_{j})^{2}|\mathcal{D}_{n}]=\hat{\sigma}_{RF}^{2}-2\epsilon_{j}\mathbb{E}[\epsilon_{j}^{*}|\mathcal{D}_{n}]+\epsilon_{j}^{2}=\hat{\sigma}_{RF}^{2}+\epsilon_{j}^{2}$ almost surely. Conditioning on $\mathcal{D}_{n}$, we know that $(\epsilon_{j}^{*}-\epsilon_{j})$ and $(\epsilon_{\ell}^{*}-\epsilon_{\ell})$ are independent for $j\neq\ell$ such that $\mathbb{E}[(\epsilon_{j}^{*}-\epsilon_{j})(\epsilon_{\ell}^{*}-\epsilon_{\ell})|\mathcal{D}_{n}]=\mathbb{E}[(\epsilon_{j}^{*}-\epsilon_{j})|\mathcal{D}_{n}]\mathbb{E}[(\epsilon_{\ell}^{*}-\epsilon_{\ell})|\mathcal{D}_{n}]=\epsilon_{j}\epsilon_{\ell}$ almost surely. Combining these two results, we obtain with (11):

[TABLE]

where the inequality results by applying (11) on the weights and the last equality from the fact that $\mathbb{E}[\epsilon_{j}\epsilon_{\ell}]=\mathbb{E}[\epsilon_{j}]\mathbb{E}[\epsilon_{\ell}]=0$, the convergence from Theorem 1 and $a_{n}^{2}/n\rightarrow 0$. Furthermore, by applying Jensen’s inequality, we obtain

[TABLE]

where the last two inequalities arise by using the identical distribution of the sequence $\{(m_{n}^{OOB}(\mathbf{X}_{i})-m(\mathbf{X}_{i}))^{2}\}_{i=1}^{n}$ and the fact that the random forest weights sum up to one. Finally, we have

[TABLE]

where the last two equalities follow from the identical distribution of $\{m_{n,b}(\mathbf{X}_{i})\}_{b=1}^{B}$ with respect to the bootstrap measure $\mathbb{P}^{*}$ and the identical distribution of $\{\mathbb{E}[(m_{n,1}^{OOB}(\mathbf{X}_{i})-m_{n}(\mathbf{X}_{i}))^{2}|\mathcal{D}_{n}]\}_{i=1}^{n}$. The convergence in (16) follows by plugging in (14) and (15) into (5).

∎

Proof of Theorem 3.

Sticking to the same notation as in Theorem 2, we obtain the following lower bound for the random forest weights:

[TABLE]

Since the prescribed parametric bootstrap approach makes use of the same tree structure as $m_{n}^{OOB}$, the bootstrapped prediction $\hat{Y}_{i,b}^{*}=m_{n,b}^{OOB}(\mathbf{X}_{i})$ can also be rewritten as

[TABLE]

As these quantities are i.i.d. in the index $b$ for fixed $i$, we can apply the strong law of large numbers conditioned on fixed $\mathcal{D}_{n}$ to obtain

[TABLE]

almost surely as $B\rightarrow\infty$ given $\mathcal{D}_{n}$. Moreover, we have

[TABLE]

Furthermore, due to the independence of the sequence $\{Y_{j}^{*}\}_{j}$ with respect to the boostrap measure $\mathbb{P}^{*}$, we obtain

[TABLE]

Combining the results from (17), (5) and (21), we obtain:

[TABLE]

where we inserted $\sum_{\ell=1}^{n}W_{n,\ell}^{(OOB)}({\bf X}_{i})=1$ (almost surely) in the third and second last step and utilized that $\mathbb{P}_{\Theta}^{2}(\mathbf{X}_{i}\stackrel{{\scriptstyle\Theta}}{{\leftrightarrow}}\mathbf{X}_{i})=1$ in the last inequality. Finally, the result follows from (19).

∎