TL;DR
This paper investigates the bias and adaptivity of CART decision trees, revealing their relationship with variable importance measures and establishing the consistency of random forests under broad conditions.
Contribution
It provides a theoretical analysis linking bias, impurity reduction, and variable importance in CART, and proves the consistency of random forests.
Findings
Decision trees with CART have small bias and are adaptive to signal strength.
The mean decrease in impurity (MDI) correlates exponentially with the probability content of terminal nodes.
Random forests are shown to be consistent under general conditions.
Abstract
Decision trees with binary splits are popularly constructed using Classification and Regression Trees (CART) methodology. For binary classification and regression models, this approach recursively divides the data into two near-homogenous daughter nodes according to a split point that maximizes the reduction in sum of squares error (the impurity) along a particular variable. This paper aims to study the bias and adaptive properties of regression trees constructed with CART. In doing so, we derive an interesting connection between the bias and the mean decrease in impurity (MDI) measure of variable importance---a tool widely used for model interpretability---defined as the sum of impurity reductions over all non-terminal nodes in the tree. In particular, we show that the probability content of a terminal subnode for a variable is small when the MDI for that variable is large and that…
| Variable | |
|---|---|
| daughter node | ||
|---|---|---|
| right | yes | |
| left | no | |
| left | yes | |
| right | no |
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsStatistical Methods and Inference · Explainable Artificial Intelligence (XAI)
Analyzing CART
Jason M. Klusowski
Department of Statistics
Rutgers University – New Brunswick
Piscataway, NJ, USA, 8019
[email protected] This research was supported in part by NSF Grant DMS-1915932.
(June 24, 2019)
Abstract
Decision trees with binary splits are popularly constructed using Classification and Regression Trees (CART) methodology. For binary classification and regression models, this approach recursively divides the data into two near-homogenous daughter nodes according to a split point that maximizes the reduction in sum of squares error (the impurity) along a particular variable. This paper aims to study the bias and adaptive properties of regression trees constructed with CART. In doing so, we derive an interesting connection between the bias and the mean decrease in impurity (MDI) measure of variable importance—a tool widely used for model interpretability—defined as the sum of impurity reductions over all non-terminal nodes in the tree. In particular, we show that the probability content of a terminal subnode for a variable is small when the MDI for that variable is large and that this relationship is exponential—confirming theoretically that decision trees with CART have small bias and are adaptive to signal strength and direction. Finally, we apply these individual tree bounds to tree ensembles and show consistency of Breiman’s random forests. The context is surprisingly general and applies to a wide variety of multivariable data generating distributions and regression functions. The main technical tool is an exact characterization of the conditional probability content of the daughter nodes arising from an optimal split, in terms of the partial dependence function and reduction in impurity.
Index terms — Decision tree, regression tree, recursive partition, CART, random forest, boosting, nonparametric regression, high-dimensional statistics
1 Introduction
Decision trees are the building blocks of some of the most important and powerful algorithms in statistical learning. For example, ensembles of decision trees are used for some bootstrap aggregated prediction rules (e.g., random forests [10]). In addition, at each iteration of gradient tree boosting (e.g., TreeBoost [18]), the pseudo-residuals are fit with decision trees as base learners. From an applied perspective, decision trees have an appealing interpretability and are accompanied by a rich set of analytic and visual diagnostic tools. These attributes make tree-based learning particularly well-suited for applied sciences and related disciplines—which may rely heavily on understanding and interpreting output from a black-box model and the system that generated the data.
Although, as with many aspects of statistical learning, good empirical performance often comes at the expense of rigor and transparency.111For a delightful read and intriguing perspective on this trade-off, see [9]. Tree-structured learning with decision trees is no exception—statistical guarantees for popular variants, i.e., those that are actually used in practice, are hard to find. The complicated recursive way in which decision trees are constructed makes them unamenable to analysis. While a complete and thorough picture of decision trees and their role in ensemble learning may be far away or even unattainable, in this paper, we take a significant step forward and aim to tackle the following three questions.
- •
Why do decision trees have small bias?
- •
Why are decision trees locally adaptive to signal strength?
- •
Can one connect bias and adaptivity with quantities used to assess variable importance?
The first two questions above are supported by an abundance of empirical evidence [12], yet remain to be explained or answered by a sensible mathematical theory, especially when the tree construction involves the input and output data. For example, decision trees are known to have small bias when they are grown deeply—a defining characteristic of random forests. But how and why? Moreover, tree-based learning is known to be particularly effective in high-dimensional sparse settings when the distribution of the output depends only on a few predictor variables. What mechanism of trees enables this?
Let us emphasize that we are not content with merely a study of the standard certificates for good predictors (e.g., consistency or rates of convergence). Rather, we aim to identity and explore the unique advantages of tree-structured learning and, in doing so, develop a unifying theory that connects bias and adaptivity via two well-known data-analytic tools for model interpretability—partial dependence functions and variable importance measures.
To make our work informative to the applied user of decision trees, we strive to make the least departure from practice and therefore focus specifically on Classification and Regression Tree (CART) [12] methodology—by far the most popular for regression and classification problems. With this methodology, the tree construction depends on both the input and output data and is therefore data-dependent. This aspect lends itself favorably to the empirical performance of CART, but poses unique mathematical challenges. As far as we know, the only other work that studies the bias of CART is [38], who used it to show asymptotic consistency of Breiman’s random forests for additive regression models. One goal of the present paper is to extend this theory to other response surfaces.
Because individual decision trees are unstable and subject to large sampling variability222This is particularly the case for classification trees., we do not concern ourselves with a study of their variance. Indeed, such an endeavor is worthwhile only in the presence of some sort of variance reduction technique like pruning (i.e., removing portions of the tree in order to reduce its complexity) or ensemble averaging (e.g., bagging [8], random forests, boosting), and even so, the bias remains the most challenging aspect of the analysis. This is, of course, not to diminish variance as an essential component of a theoretical investigation. For instance, in the sequel, we will apply our individual tree bias bounds to Breiman’s random forests (which use ensembles of trees) in conjunction with existing results for its variance.
Let us now describe the statistical setting and framework that we will operate under for the rest of the paper. For clarity and ease of exposition, we focus specifically on regression trees, where the target outcome is a continuous real value. Although, many of our results hold for binary classification trees as well.
We assume the learning (training) data is , where , are i.i.d. with common joint distribution and joint density with respect to Lebesgue measure (with marginal distribution and density, and , defined analogously). Here, is the input (feature, covariate, or predictor vector) and is a continuous outcome (response or output variable). A generic pair of variables will be denoted as . A generic coordinate of will be denoted by , unless there is a need to highlight the dependence on the coordinate index, say , where . We will use the terms feature, predictor, or input variable to refer to interchangeably. The statistical model is , for , where is an unknown regression function and are i.i.d. errors. The conditional average of given is optimal for prediction if one uses squared error loss since it minimizes the conditional risk . The bias of a prediction rule at a point is henceforth defined to be .
One key strength of decision trees is that they can exploit, if present, low local dimensionality of the response surface. This is particularly useful since many real-world input/output systems admit or are approximated well by local sparse representations of simple model forms, e.g., wavelets and neural networks. That is, even though the input/output relationship is determined by a large number of variables overall, the dependence may be locally characterized by only a small subset of variables. Such local adaptivity is made possible by the recursive partitioning of the input space, in which optimal splits are increasingly affected by local qualities of the data as the tree is grown. Hence, variables that locally have more influence on determining the response are much more likely to be included as candidates for further splitting. In essence, decision trees have a built-in local variable subset selection mechanism, which allows them to overcome many of the undesirable consequences (e.g., overfitting and large sample requirements) of high-dimensional modeling.
In this paper, we shall work with a more simplistic, yet still exemplary model and assume that the conditional mean response depends only on a small, unknown subset of the features. In other words, is almost surely equal to its restriction to the subspace of its “strong” variables , where . Conversely, the output of does not dependent on “weak” variables that belong to . Of course, the set is not known a priori and must be learned from the data.
2 Organization
This paper is organized according to the following schema. In Section 3, we establish some basic notation and definitions that we will use throughout the paper. Section 4 reviews some of the terminology and quantities associated with growing regression trees. In Section 5, we review two important data-analytic quantities associated with tree-ensembles for visualization and interpretability. A summary of the main results is given in Section 6, including some accompanying examples and simulations studies. Section 7 discusses the main assumptions on the regression function that we use to obtain our bounds. In Section 8, we apply our bounds for individual decision trees from Section 6 to show consistency of Breiman’s random forests. Section 9 contains some finite sample results. Finally, in Section 10, we briefly discuss how our results can be extended to binary classification. Proofs of the main statements from Section 6 are given in Section 11 and Appendix A contains proofs of some lemmas and examples from the body of the paper.
3 Notation and definitions
For , let . With a slight abuse of notation, we write instead of for brevity. If is a subset of , we let and . For two subsets , . For and , we define , where .
For a function and subset , we define the oscillation of on by . The total variation of a function on an interval is defined by , where . If , we will write to denote the order partial derivative of with respect to the variable at the point . If , we will write to denote the first derivative of at the point . The order derivative of at the point is denoted by .
4 Preliminaries
As mentioned earlier, regression trees are commonly constructed with Classification and Regression Tree (CART) [12] methodology. The primary objective of CART is to find partitions of the input variables that produce minimal variance of the response values (i.e., minimal sum of squares error with respect to the average response values). Because of the computational infeasibility of choosing the best overall partition, CART trees are greedily grown with a procedure in which binary splits recursively partition the tree into near-homogeneous terminal nodes. That is, an effective binary split partitions the data from the parent tree node into two daughter nodes so that the resultant homogeneity of the daughter nodes, as measured through their impurity (within node sum of squares error), is improved from the homogeneity of the parent node. Under the least squares error criterion, it can easily be shown that if one desires to have constant output in the terminal nodes of the tree, then the constant to use in each terminal node should be the average of the response values within the terminal node [12, Proposition 8.10]. Hence, these models produce a histogram estimate of the regression surface. As such, they are often referred to as piecewise constant regression models, since the tree output is constant on each terminal node.
Let us now describe the algorithm with additional precision. Consider splitting a regression tree at a node . Let be a candidate split for a variable that splits into left and right daughter nodes and according to whether or . These two nodes will be denoted by and . As mentioned previously, a tree is grown by recursively reducing node impurity, which, for regression trees grown with CART, is determined by within node sample variance
[TABLE]
where is the sample mean for and is the number of observations in . Similarly, the within sample variance for a daughter node is
[TABLE]
where is the sample mean for and is the sample size of (similar definitions apply to ). The parent node is split into two daughter nodes using the variable and split point producing the largest decrease in impurity (or impurity reduction). For a candidate split for , this decrease in impurity equals [12, Definition 8.13]
[TABLE]
where and are the proportions of observations in that are contained in and , respectively. Note that maximizing is also equivalent to minimizing
[TABLE]
which means that CART seeks the split point that minimizes the weighted sample variance. Yet another way to view is via its equivalent representation , where is the empirical correlation between and the decision stump within . Hence, at each node, CART seeks the step function most correlated (in magnitude) with the response variable.
To reiterate, the tree is grown recursively by finding the split point that maximizes . The particular variable chosen is the one that gives the largest reduction in impurity over . We denote an optimized split point by (breaking ties arbitrarily) and the optimally split daughter nodes with by and , i.e., and , respectively. The tree output at a terminal node is .
The use of (2) to evaluate each candidate split involves several passes over the training data with the consequent computational costs when handling problems with a large number of predictor variables and observations. This is particularly serious in the present setting of continuous variables—the major computational bottleneck of growing trees. Fortunately, the use of the least squares error criterion with averages in the terminal nodes permits further simplifications of the formulas described above. That is, using the sum of squares decomposition, can equivalently be expressed as [12, Section 9.3]
[TABLE]
This expression implies that one can find the best split for a continuous variable with just a single pass over the data, without the need to calculate multiple averages and sums of squared differences for these averages. In fact, not only does this representation have computational benefits—it will also be crucial in establishing our main results. It should be stressed that this alternative expression is unique to the least squares error criterion with averages in the terminal nodes of the tree.
A direct analysis of regression trees using the finite sample splitting criterion is challenging and obfuscates some of the inner and subtle mechanisms that makes tree-based learning with CART desirable. Instead, to remain true to the original CART procedure while still being able theoretically study its dynamics, we work under an asymptotic data setting for determining splits and therefore replace with its analog based on population parameters:
[TABLE]
where is the conditional population variance , and are the daughter conditional variances
[TABLE]
and and are the conditional probabilities (probability content) of the daughter nodes arising from the split , i.e.,
[TABLE]
These conditional probabilities can also be interpreted as the infinite sample proportion of observations in the parent node that are contained in the daughter nodes. Note that is also the distribution function of . We will occasionally write instead of to highlight dependence on the split . The density function of , that is, , will be denoted by or . The infinite sample analog of (4) is
[TABLE]
We let and denote the optimally split daughter nodes with optimal split , respectively. Note that any node is a Cartesian product of intervals, which we call subnodes. The subnode of variable within node is denoted by , where .
Under our framework, we optimize the infinite sample splitting criterion, namely, , instead of the empirical one (2). Optimizing the splitting protocol can also be viewed as querying an oracle for the optimal (population) split values. A maximizer of is denoted by , i.e., (again, breaking ties arbitrarily). Despite this tweak, we stress that only the splits are determined from an infinite sample quantity; all other aspects of the regression tree (e.g., terminal node values, depth, variables selected for candidate splits) are determined from the learning sample. More specifically, the regression tree still outputs , except that is determined by an infinite sample (population) objective. If the number of observations within is large and has a unique global maximum, then we can expect (via an empirical process argument) and hence our infinite sample setting is a good approximation to CART with empirical splits. Indeed, if is unique, [3] and [13, Section 3.4.2] show cube root asymptotics (i.e., converges in distribution) of split points for multi-level decision trees using the CART sum of squares error criterion. While these results show that and are close, they do not reveal what sort of partition of the input space is induced by a sequence of optimal splits—the goal of the present paper. Since this partition is constructed from the distribution of , we will need to impose conditions on the regression function and input distribution so that the partition yields a predictor with small bias. It is our hope that the applied user of decision trees can benefit from knowing these limitations in the idealized setting of population-level splits.
Since the recursive partition obtained from governs the bias of the tree, our study of the partitions created from the infinite sample version is in much the same vein as the study of kernel or nearest neighbors regression with (oracle) infinite sample, optimal parameters (e.g., kernel bandwidth or number of nearest neighbors), obtained by minimizing the asymptotic mean integrated squared error (AMISE).
4.1 A comment on notation
Up until now, we have assumed that the split occurs along a generic coordinate . However, in the multi-dimensional setting, there will be a need to specify the coordinate index. Therefore, we will write to denote when the coordinate being split is . For the same reason, we write and in lieu of and , respectively. Similar definitions will hold for the subnode when we want to emphasize the dependence on the variable index. We write to denote the splitting variable chosen in node , i.e., , breaking ties arbitrarily. Finally, note that an optimal split for a node depends on and therefore should technically be written as , however, we suppress this dependence for brevity and assume that it holds implicitly. We also use similar notation for the finite sample versions of these quantities. As a general rule, if the index is omitted on any quantity, it should be understood that we are considering a generic variable .
We now survey two popular data-analytic quantities that are also intimately connected to the forthcoming results in Section 6.
5 Data-analytic quantities for interpretation and visualization
Pertinent to many tasks in data analysis is interpretability of the representation of the model. For black-box models such as random forests, this is usually performed through graphical renderings of the prediction surface as a function of one or two predictor variables and assessing the role each variable plays in determining the final output. Here we review two of the most popular quantities associated with decision tree ensembles for this purpose.
5.1 Partial dependence functions
One useful device for visualizing the influence of a particular variable on the output is the so-called partial dependence function.333Visualizing the effect of more than two predictor variables has limited descriptive value. By looking simultaneously at the partial dependence plots—i.e., a trellis of plots of the partial dependence functions for each variable—one can obtain a visual description of a multivariable predictor. In the same spirit, for assessing how each variable affects the output conditional on a node, we define the conditional partial dependence function given node via
[TABLE]
where is the conditional probability measure with conditional density . Note that is the best least squares approximation to in as a function of alone, i.e., . For brevity, we write to denote the derivative , provided it exists. We also define the mean-centered conditional partial dependence function as
[TABLE]
that is, is minus its mean value over the node.
Strictly speaking, is not the same as the partial dependence function in the sense of [18, Section 8.2] or [24, Section 10.13.2], where, instead of ignoring the effects of , one looks at the dependence of on in after averaging with respect to the marginal distribution of the other covariates , i.e., , where is the conditional probability measure with conditional density . However, when and are independent (for example, as in the uniform case), the two definitions coincide with each other.
Of course, in practice, one would integrate an estimate of against the empirical distribution, so that, for example,
[TABLE]
When the tree is fully grown so that each terminal node contains only a single observation, i.e., for all , (8) is the Individual Conditional Expectation (ICE) [21].
Let us finally mention that a desirable property of tree-based models (i.e., if is a tree-structured predictor) is that the partial dependence function (8) can be quickly computed from the tree itself, without passing over the data each time it is to be evaluated. This fact has implications for the rapid production of a trellis of plots for each variable.
5.2 Variable importance measures and variable influence ranking
Another attractive feature of tree-based ensembles is that one can compute, essentially for free, various measures of variable importance (or influence) using the optimal split points and their corresponding impurities. These measures are used for sensitivity analysis and to rank the influence of each variable in determining the output, which in turn, can be used to identify and select the most relevant predictors for further investigation (such as plotting their partial dependence functions). For random forests, one canonical and widely used measure is the Mean Decrease in Impurity (MDI) [18, Section 8.1], [12, Section 5.3.4], [24, Sections 10.13.1 & 15.3.2], defined as the weighted sum of largest impurity reductions (i.e., either Gini impurity for classification or variance impurity for regression) over all non-terminal nodes in the tree, averaged over all trees in the forest, i.e.,
[TABLE]
where
[TABLE]
and the sum extends over all non-terminal (internal) nodes such that and is the proportion of observations that land in node .444Another commonly used and possibly more accurate measure is Mean Decrease in Accuracy (MDA), defined as the average difference in out-of-bag error before and after randomly permuting the values of in out-of-bag samples over all trees. However, as with all permutation based methods, computational issues are present. Compare MDA with MDI, which can be computed as each tree is grown with no additional cost. Both MDI and MDA are calculated in the R package randomForest with randomForest(..., importance = TRUE) and in the Python library scikit-learn with attribute feature_importances_
The next definition of variable importance generalizes and localizes MDI. Instead of weighting each by the proportion of observations in , more flexible weights are allowed. Moreover, rather than summing over all non-terminal nodes, we condition on a particular terminal node so that only nodes that are ancestors of the terminal node are considered.
Definition 1** (Conditional variable importance).**
Let be a terminal node and let denote the index of the variable selected to split along at an ancestor node of . The conditional variable importance of given is defined by
[TABLE]
where the sum extends over all ancestor nodes of such that and the (possibly data-dependent) weights are nonnegative. That is, is a weighted sum of largest impurity decreases among all ancestor nodes of terminal node such that was selected for a split. The infinite sample version of , denoted by , is defined by
[TABLE]
where the weights are nonnegative.
Note that is a global measure of importance. Thus, is relevant for answering questions such as “what variables are most important in predicting blood pressure?” and is useful for answering questions like “for subjects ages 65 and up, what variables are most important in predicting blood pressure?”. A few other variants of local or case-wise (permutation) importance have been proposed to cope with the bias of variable importance measures towards correlated predictor variables. See, for example, [41].
6 Summary of results
This section contains our main results. Due to space constraints, proofs of most of the statements are deferred until Section 11 and Appendix A.
6.1 Distributional assumptions
Before we continue, let us first state the main distributional assumptions on the predictor variables. Let us remark, however, that many of the subsequent statements and results (e.g., Section 8, Section 9, and Section 11) hold without them.
Assumption 1**.**
For each variable and node , the distribution function is strictly increasing.
Assumption 1 is quite mild and holds if the joint density never vanishes.555This does not necessarily mean that the joint density is uniformly bounded below by a positive constant.
The next assumption is needed for Theorem 1, our main result.
Assumption 2**.**
For each variable , node , and split ,
[TABLE]
for some universal constant .
Assumption 2 is more restrictive than Assumption 1, yet it still allows for some degree of dependency structures among the predictor variables. For example, it holds if the joint density function of the features is uniformly bounded above and below by constant multiples of the product of its marginal densities, i.e.,
[TABLE]
for all , where and are positive constants. In this case, can be taken to be .
6.2 Probability content of terminal subnodes
The next theorem, our main result, gives a clean, interpretable bound on the -probability of a terminal subnode in terms of the variable importance measure, which in turn, controls the bias of the tree. Specifically, it shows that conditional terminal subnode probability content is (exponentially) small in the importance measure attributed to the splitting variable. This bound also corroborates with empirical evidence showing that, although decision trees are highly unstable, their bias tends to be small. For brevity, we furnish the proof in Section 11.
Theorem 1**.**
Suppose Assumption 1 and Assumption 2 hold and is a subnode along the direction for a terminal node . Then,
[TABLE]
where is the conditional variable importance (11) with weights given by
[TABLE]
Furthermore, if the first-order partial derivatives of the regression function and joint density function of exist and are continuous, then
[TABLE]
The main technical tools for proving Theorem 1—Theorem 7 and Theorem 9—characterize in terms of the partial dependence function and reduction in impurity at an optimal split. The form of the weights (15) are derived from the first-order conditions (i.e., ) of as a global maximizer of , whereas the lower bound (16) incorporates both first- and second-order optimality conditions (i.e., and ).
Remark 1**.**
If the regression function is linear and the input is uniformly distributed, then with weights (15) equals , the number of times was selected among all ancestor nodes of terminal node .666Importance measures based on feature selection frequencies are implemented in standard R and Python packages for XGBoost [14].
Remark 2**.**
Theorem 1 implies that converges to zero in -probability when for each . According to classic theory for partitioning-based prediction rules, shrinking terminal node diameters is a necessary condition for asymptotic consistency [40].
In light of (14), it is tantalizing to interpret (and therefore ) as truly a measure of variable importance, since it governs the bias of the regression tree along a specific direction, i.e., the conditional probability content of a terminal subnode for a variable is small when the MDI for that variable is large.777The original motivation for MDI was based solely on simple heuristic arguments and so one is left guessing as to why it seems to work. Overall, little is known about the theoretical properties of variable importance measures; see [26, 30, 28] for recent attempts in this direction.
As a consequence of Theorem 1, we immediately see two important properties of the regression tree:
- •
Terminal subnodes with smaller -probability are along more “important” directions as delineated by MDI, and thus adapt to signal strength in each direction. That is, the tree needs to be split more often in order to create finer partitions along directions that are more relevant to the output.
- •
The adaptation to the importance of the variable is local and depends on a particular terminal node of the tree. That is, each direction and location of the features requires a different level of granularity in the tree in order to detect and adapt to local changes in the regression surface. For example, regions of the input space where the response is “flat” do not need to be split as often and therefore a crude partition will suffice. On the other hand, complex functional dependencies may require a higher degree of fineness in the final model.
These observations are consistent with [29, Section 4], in that terminal nodes are on average narrower in directions with strong signals than in directions with weak signals. This is a very appealing property from a statistical perspective—trees with terminal subnodes that have large (resp. small) probability content in directions with weak (resp. strong) signals are less likely to overfit (resp. underfit) the data.
Remark 3**.**
An analogy can be drawn between the terminal subnode probability content and the bandwidth size in kernel regression, both of which control the bias of the predictor. Bandwidth selection for kernel regression [48] is often performed using a plug-in estimator of the best bandwidth that minimizes the AMISE. There, just like with (16), the best theoretical bandwidth also depends on the density function of the features and the smoothness of the regression function (albeit, via curvature—) [48, Theorem 1] and [47, Section 5.8].
Remark 4**.**
As will be discussed in Section 10, the statement of Theorem 1 also holds verbatim for binary classification trees with appropriate modifications for alternate (i.e., Gini) splitting rules. An illustrative example involving logistic regression will also be provided.
6.3 Empirical study
In Fig. 1, we showcase the aforementioned adaptive properties of trees on four representative datasets from the UC Irvine Machine Learning Repository, namely, the Airfoil Self-Noise (Fig. 1(a)) and Concrete Compressive Strength (Fig. 1(b)) datasets for regression and the Blood Transfusion Service Center (Fig. 1(c)), and HTRU2 (Fig. 1(d)) datasets for binary classification. We first standardized the input data so that each variable belongs to , i.e., . For each dataset, we generated trees from bootstrap samples of the data. Each tree was generated by rpart in R with default settings, except for the complexity parameter cp, which was set to . The black bars represent the median terminal subnode length for each tree, averaged over all trees. The white bars represent (9) for each variable. Both quantities are scaled so that the largest among them is and the variables are ordered according to increasing . In agreement with Theorem 1, the barplots reveal the inverse relationship between and the terminal subnode lengths .
6.4 Lower bounds on MDI
In light of Theorem 1 and Remark 2, it is natural to ask when diverges. We now provide some answers. First, let us mention that studying directly is hopeless since it is nearly impossible to give a closed form expression for each . Nevertheless, can still be lower bounded by giving a lower bound on each weight and reduction in impurity .
By definition of , one can lower bound each by for any . Effectively, this means that to lower bound , one can replace by for any choice in or (per a Bayesian perspective) by the integrated decrease in impurity with respect to a prior on the splits. (It is often convenient to choose to be the median of .) This observation is crucial to the forthcoming analysis, since it reduces the burden of finding exactly to finding a suitable choice of or prior for which is tractable to analyze.
To lower bound the weights in Theorem 1, one is confronted with obtaining a useful upper bound on either for (15) or for (16). We will see that it typically suffices to bound either by their supremum norm over splits in the parent subnode, and so no explicit knowledge of is required. For the lower bound (16), one additionally needs to lower bound the conditional density of at , or . This too is a simple task if the joint density of is uniformly bounded away from zero by a positive constant, in which case . For example, if is uniformly distributed, then .
Using these observations, we show in Theorem 2 that can be lower bounded by a positive constant multiple of the selection frequency of in the tree. For brevity, we defer its proof until Section 11. Before we state Theorem 2, we first introduce some concepts. Central to the paper is a quantity which we call the node balancedness. It measures the infinite sample proportion of data in the parent node that is contained in either daughter node from an optimal split.
Definition 2** (Node balancedness).**
The balancedness of a node is defined by
[TABLE]
Another way of thinking about node balancedness is the following. Suppose we randomly generate a new from and classify if or if . Then is simply the variance of .
The node balancedness is always one (perfectly balanced) when the split is performed at the median of the conditional distribution . This particular situation occurs in the special case that the regression surface is linear and the input distribution is uniform. In general, the quantity depends on the node —if changes, so does . We now introduce a more global measure, which depends only on the regression function.
Definition 3** (Global balancedness).**
The global balancedness is defined as
[TABLE]
*where the infimum runs over all parent nodes of the best split left and right daughter nodes and , respectively. *
With these definitions in place, we are now ready to state Theorem 2, which lower bounds in terms of the selection frequency for . For brevity, the proof is deferred until Section 11.2.
Theorem 2**.**
Suppose the direction of is not too “flat” in the sense that there exists a finite integer such that for each in , there is a finite-order partial derivative, with , that is nonzero and continuous for all other input coordinates in . More formally, assume
[TABLE]
is finite. If additionally the features of are independent and have marginal densities that are continuous and never vanish, then the global balancedness is strictly positive and
[TABLE]
where is the number of times was selected among all ancestor nodes of terminal node .
It will be shown in Section 7 (see Theorem 3) that any linear combination of Gaussian radial basis functions in with positive weights satisfies (17). Furthermore, (17) also holds for any nonconstant, one-dimensional polynomial or partial sum of a Fourier series.
Remark 5**.**
Taken together, Theorem 1 and Theorem 2 imply that converges to zero (exponentially fast) in -probability when for each . The feature selection frequencies of important variables are typically large for deeply grown decision trees (in fact, is usually scales with the tree depth) and hence this condition is typically met. That is, the number of times at an ancestor node of typically scales with the tree depth.
Remark 6**.**
Condition (17) does not mean that all partial derivatives of exist and are continuous. For example, the function has discontinuous second derivative, yet still satisfies the condition.
6.5 Examples
Theorem 2 does does not show how depends on the structure of the regression function. It seems, at least for now, that such results are only available on a case-by-case basis and obtained with considerable effort. Here we give some example calculations of for polynomial and trigonometric functions which decay inversely with the degree and periodicity, respectively. These theoretical results are accompanied by plots (see Fig. 2 and Fig. 3) of together with sampling distributions of from a sample size of over independent replications. Note that here a smaller sample size was purposely chosen to mimic a situation where the split is performed in a deep node and thus likely to contain only a small number of observations. As evidenced by the plots, the optimal splits tend to be closer to the parent subnode edges (in this case [math] and ) with larger degree and periodicity. This phenomenon is manifested in the lower bounds on in Example 1 and Example 2—some of the terminal nodes will be large if splits from ancestor nodes are close to their parent subnode edges. In conjunction with Theorem 1 and the inverse relationship between the terminal node size and (being a weighted sum of ), the plots Fig. 2 and Fig. 3 also reveal that the splits tend to be near the edges when the reduction in impurity is small. In future sections, we will theoretically confirm this (see Theorem 7 and Theorem 9) and show, more generally, that splits occur near the edges of the parent subnode whenever is small.888Note that Theorem 7 and Theorem 9, used to prove this phenomenon, do not need Assumption 2. This phenomenon has also been dubbed “end-cut preference” in the literature [25], [12, Section 11.8]. In Section 11.1, we will study a penalized variant of in order to mitigate this effect.
For each of the following three examples, we assume that is uniformly distributed on .
Example 1**.**
Suppose for nonzero constants and integer . Suppose so that . Then
[TABLE]
It is possible to show that, more generally, if , for constants , then for some universal constant .
Example 2**.**
Suppose for nonzero constants and integer . Suppose so that . There exists a universal constant such that
[TABLE]
Example 2 implies that to ensure each terminal subnode has small probability content, (or the tree depth) should be larger for functions that have larger frequencies in their frequency domain. This is not merely a coincidence. In fact, it can be shown more generally that if is uniformly distributed, then , where are the Fourier coefficients of the conditional partial dependence function . This can easily be established by lower bounding by , where is the uniform prior on , and using Parseval’s identity. That is,
[TABLE]
Our final example is a quintessential non-additive regression function known as “Friedman #1” from [17, Section 4.3]. This function was used by Friedman to illustrate the efficacy of Multivariate Adaptive Regression Splines (MARS).
Example 3**.**
Suppose
[TABLE]
Then each is bounded from below in Table 1.
7 Assumptions on the regression function
Note that variable “irrelevance” is not the same as conditional independence, i.e., , as some variable can be irrelevant in the sense that , yet it still influences the distribution of output values, i.e., .999See [30] for more details along these lines. Therefore, in order to ensure that is large for all nodes in direction with a truly strong signal, i.e., , we need a condition like (17) so that the regression function is marginally not too “flat” and hence .101010Conversely, for may be on the same order as for due to spurious correlation between and . Thus, the CART algorithm may have a selection bias and spend unnecessary time splitting on , when conditional on , this variable plays no role in determining the output, i.e., . This situation is less troublesome from an approximation error perspective, but becomes highly relevant for variable selection and interpretability. Condition (17) is sufficient (but not necessary111111For example, in one dimension, there are examples of regression functions whose order derivatives in (17) have jump or removable discontinuities, yet .) for the existence of a positive which depends only on the regression function. In the additive case, i.e., when , condition (17) is stronger than saying that each is continuous and nonconstant on each subnode. For example, flat functions, i.e., with for all , or functions with derivatives having essential discontinuities, i.e., may be both continuous and nonconstant on each subnode, yet violate (17). In particular, for both functions, for all nodes—however, must equal zero.
In the multivariable, non-additive case, the function does not satisfy (17) because and are both equal to and therefore are constant. In fact, if is uniformly distributed and , then integrating out either direction yields a constant function on and hence for . More generally, if for , then any split along any variable in results in a zero decrease in impurity [39, Technical Lemma 1], despite the possibility that the regression function is nonconstant on . Such a situation may lead one to erroneously classify certain features as “weak” when they may not be so. Practically speaking, this means that the CART algorithm may ignore certain variables and therefore fail to create a fine enough partition of , which may introduce a large amount of bias. Consider again the example with uniformly distributed predictors and suppose we wish to split at node . Any further splitting via the CART protocol will result in zero impurity reduction. Thus, one may be tempted to assume the function is constant on the node when in fact “strongly” depends on both variables. Hence, we require some sort of “self-consistency” property of the regression function, namely, if is zero for all splits for , then is constant on and therefore the bias of the tree on that node is zero. With this self-consistency assumption, even though the diameters of the nodes may not converge to zero, for the purpose of controlling the approximation error, one can safely ignore the regression function entirely on that node, regardless of whether the algorithm actually performs any further splitting.
Despite the aforementioned difficulties, we show in the sequel that any linear combination of Gaussian radial basis functions with positive weights satisfies (17). To this end, consider the function class
[TABLE]
We allow for the possibility that contains diagonal entries that are equal to zero, with the interpretation that the corresponding term is independent (constant) in that coordinate. It is known that is a dense subclass of all positive continuous functions on [33]. We have the following theorem.
Theorem 3**.**
If , then any satisfies (17).
Proof.
Without loss of generality, suppose each weight of combination is strictly positive. Then as a function of has the form of a one-dimensional combination of Gaussian radial basis functions with positive weights, i.e., where , , and belong to . It can further be assumed without loss of generality that each Gaussian function in the sum is distinct. Suppose, contrary to hypothesis, that does not satisfy (17). Then, since has continuous partial derivatives of all orders, there exists such that for all . However, as a function of is analytic and so is constant in a neighborhood of , say . This implies that the partial derivative of with respect to at , namely , is equal to zero for all . Thus, one is confronted with the question about the multiplicity of ways to represent the zero function of an exponential polynomial. To answer this, consider the following lemma, which is a special case of a much more general result in complex analysis [27, Theorem 1.6], [22] and can be deduced using induction and differentiation. For the sake of completeness, we include its proof in Appendix A.
Lemma 1** (Linear independence of exponential polynomials).**
Suppose are distinct (real or complex) polynomials without constant terms and are (real or complex) polynomials. If on an open subset (of the reals or complex plane), then .
Note that is an exponential polynomial of the form and hence by Lemma 1 above, , implying that for all (since ). Thus is constant in on for all , which is a contradiction to the assumption that . ∎
Remark 7**.**
A similar argument to Theorem 3 can be used to show that any nonconstant polynomial or partial sum of a Fourier series in also satisfies (17). Hence, there is a dense collection of additive regression functions that satisfies (17).
Individual decision trees are not competitive predictors, since their high variability and tendency to overfit makes them generalize poorly to new data. Random forests, on the other hand, are an archetypal example of variance reduction via ensemble averaging, where many weak predictors (such as decision trees) are combined to form a stronger predictor. Next, we will use Theorem 1 and Theorem 2 to show asymptotic consistency of Breiman’s random forests grown with the infinite sample CART criterion.
8 Application to random forests
Random forests are ubiquitous among ensemble averaging algorithms because of their ability to reduce overfitting, handle high-dimensional sparse settings, and efficient implementation. Due to these attractive features, they have been widely adopted and applied to various prediction and classification problems, such as those encountered in bioinformatics and computer vision. The base learner for a random forest is a binary tree constructed using the methodology of CART. Naturally, some of our analysis for decision trees can be carried over to random forests. We explore this connection in this section.
Random forests grow an ensemble of ntree regression trees.121212In what follows, we use typewriter fonts for the variable names in the R package randomForest. Each tree is grown independently using a bootstrap sample of the original data (also known as a bagged decision tree). As with traditional decision trees, terminal nodes of the tree consist of the predicted values which are then aggregated by averaging to obtain the random forest predictor. Unlike CART decision trees, random forest trees are grown nondeterministically with two levels of randomization. In addition to the randomization introduced by growing the tree using a bootstrap sample, a second layer of randomization is injected with a random feature selection mechanism. Here, instead of splitting a tree node using all features, the random forest algorithm selects, at each node of each tree, a random subset of mtry potential variables that are used to further refine the tree node by splitting. The number of potential variables mtry is often much smaller than ; for regression, the default value is . This two-level randomization is designed to decorrelate trees and therefore reduce variance. To reduce bias, random forest trees are grown deeply—in fact, each tree is grown as deeply as possible with the stipulation that each terminal node contains at least nodesize observations.
More concretely, a random forest is a predictor that is built from an ensemble of randomized base regression trees . The sequence consists of i.i.d. realizations of a random variable , which governs the probabilistic mechanism that builds each tree. These individual random trees are aggregated to form the final output
[TABLE]
When ntree is large, the law of large numbers justifies using
[TABLE]
in lieu of (19), where denotes expectation with respect to , conditionally on . We shall henceforth work with these infinite sample versions (i.e., infinite number of trees) of their empirical counterparts (i.e., finite number of trees).
Let us now briefly describe the random feature mechanism of random forests in greater detail. To ensure that candidate strong (resp. weak) coordinates have high (resp. low) selection probabilities, at each step, we randomly select without replacement a subset of cardinality mtry and then select the variable in and corresponding split that most decreases impurity within the current node. That is, for each coordinate in , calculate a split that maximizes and store the corresponding maximum value . Finally, select one variable at random among the corresponding largest elements of to further split along within the current node. For a more detailed discussion of the algorithm, see [38]. As is argued in [5, Section 3], this random feature selection procedure will produce selection probabilities that concentrate around for and zero otherwise. Hence each “strong” variable has roughly an equal chance of being selected among all “strong” variables.
Researchers have spent a great deal of effort in understanding theoretical properties of various streamlined versions of Breiman’s original algorithm [19, 20, 2, 7, 16, 5, 37]. See [6] for a comprehensive overview of current theoretical and practical understanding. Unlike Breiman’s CART algorithm, these stylized versions are typically analyzed under the assumption that the probabilistic mechanism that governs the construction of each tree does not depend on the pair (i.e., the splits to not depend on the data distribution), largely with the intent of reducing the complexity of their theoretical analysis. Such models are referred to as “purely random forests” [20]. In one variant known as a “centered random forest” (proposed by Breiman himself in a technical report [10] and later studied by [5]), the splits are performed at the midpoint of each subnode and hence corresponds to a special case of the present paper, where the input distribution is uniform and the regression surface is linear.
Inspired by the results of Theorem 1 and [37, Theorem 4.1], we now show asymptotic consistency for Breiman’s random forests with splits determined by the infinite sample CART sum of squares error criterion. Consistency of random forests with CART (albeit, with the finite sample splitting criterion) was previously only known when the regression function has an additive structure [38]. While this important work provides insight into the complicated and subtle mechanisms of random forests, it still does not adequately explain its potential as a general nonparametric method. For example, additive models are not flexible enough to allow for interactions among covariates (which limits their flexibility for multi-dimensional statistical modeling), and there are already other highly effective training algorithms such as backfitting [11].
We follow the terminology of Scornet [37] and call a random forest “totally nonadaptive” if it is built independently of the training set . Let maxnodes denote the maximum number of terminal nodes in the tree built with randomness . Due to space constraints, we defer the proof of the next result, Theorem 4, until Appendix A. For the statement of Theorem 4, we let denote the conditional mean decrease in impurity (11) with weights from Theorem 1.
Theorem 4**.**
Consider a totally nonadaptive forest predictor , where each tree is grown with the infinite sample CART sum of squares error criterion (3). Suppose that
- (a)
* is continuous on ; * 2. (b)
* with -probability one; and* 3. (c)
* with -probability one for all .*
Then as .
Remark 8**.**
If additionally the regression function satisfies the assumptions of Theorem 2, then , where . By the previous discussion, for each nonterminal node and hence goes to infinity with the tree depth -almost surely.
Remark 9**.**
When coupled with a study of the variance of the forest, our theory for the bias of individual trees (Theorem 1 and Theorem 2) enables one to determine the quality of convergence of as a function of the parameters of the random forest, e.g., sample size, dimension, sparsity level, and depth to which the individual trees are grown. The analysis also reveals a local bias-variance tradeoff, which highlights the local adaptivity of random forests. However, we shall choose not to pursue this analysis in the present paper and leave it for future work.
9 Finite sample analysis
From the perspective of studying the theoretical properties of decision trees, it is desirable if the splits do not separate a small fraction of data points from the rest of the sample. That is, the node counts should be large enough to contain enough data points so that local estimation valid. On the other hand, the node sizes should be small enough to identify local changes in the regression surface. This is true, for example, if the splitting criterion encourages splits that are performed away from the parent subnode edges. Perhaps the earliest mention of such a condition is [12, Section 12.2], where, en route to establishing asymptotic consistency, it is assumed that the proportion of data points (from a learning sample of size ) in each terminal node is at least for some sequence tending to infinity and that the diameters of the terminal nodes converge to zero in probability. A similar assumption is explicitly made in the analysis of [32] and [46] to ensure that terminal node diameters of the forest tend to zero as the sample size tends to infinity, which as mentioned earlier, is a necessary condition to prove the consistency of partitioning estimates [40], [23, Chapter 4]. This property is also satisfied by quantile forests [37], where each split contains at least a fraction of the observations falling into the parent node. Furthermore, in a Bayesian setting, comparable regularity conditions for partitions have been assumed for theoretical analysis of Bayesian random forests (e.g., BART [15]) [36, Definition 3.1], [42, Definition 2.4], [35, Definition 6.1] (e.g., in so-called median or - tree partitions [4], each split roughly halves the number of data points inside the node).
From the perspective of adaptive estimation, forcing the recursive partitions to artificially separate a fixed fraction of the data points at each step may be undesirable. Indeed, [25] argues that CART posses a desirable trait of splitting near the edges along noisy variables. This perspective, together with the fact that one does not know a priori which variables are important, leads one to the conclude that it is undesirable to sacrifice the data-dependent nature of the split criterion in order to ensure that a technical condition is met. There are currently no results stating that splits in CART are performed away from the parent subnode edges. However, our results show that the standard assumptions for “valid partitions” (for example, see [32, Assumption 3], [46, Definition 1], or [45, Definition 4]) are satisfied for CART.
In the next subsection, we show how one can go from bounds on the conditional probability content of an optimally split subnode to bounds on the distance of an optimal split to its parent subnode edges.
9.1 From probabilities to distances
Let be the quantile function of the probability measure on with distribution function , i.e., , . Suppose for the moment that . Then and hence . From this, one can deduce that lies between and , where . This allows us to go from probability estimates to distance estimates. Inspired by this observation, we impose a general condition on the form of the quantile function so that similar conclusions can be made for other input distributions. The assumption can also be interpreted as a regularity condition on . Roughly, it says that cannot be too “flat” when is near zero or one.
Assumption 3**.**
There exist two increasing bijections and such that for all nodes ,
[TABLE]
Note that and are both necessarily continuous. This device allows us to bound the distance of to the edges and in terms of the global balancedness . To see this, note that by Definition 3, and hence both and are between . Let so that, by (73),
[TABLE]
Finally, using (20), we have that
[TABLE]
and, by rearranging terms,
[TABLE]
where . These bounds show how far (in terms of distance) any optimal split is from the parent subnode edges. The inequality in (21) bounds the distance between any optimal split and the midpoint of the parent subnode.
We now provide some examples of joint distributions that satisfy Assumption 3. The proofs of each are given in Appendix A.
- (a)
Independent joint density bounded from above and below. If each is i.i.d. , then and . 2. (b)
Independent joint density unbounded from below. If each is i.i.d. , then , , and . 3. (c)
Independent joint density unbounded from above. If each is i.i.d. , then , , and .
9.2 Number of observations in nodes
In this section, we give lower bound on (a) the number of observations contained in a daughter node from an optimally split parent node using the finite sample criterion (denoted by and ) and (b) the distance of the optimal split to the edges of its parent node.131313The terminal node counts and are also called nodesize in the R package randomForest. Throughout this section, we implicitly assume that Assumption 1 and Assumption 3 hold for each of the covariates. For simplicity, we assume that is equal to the full set of variables, although under Assumption 2, one can also develop lower bounds for the number of observations that land in nodes along informative directions in , i.e., , where and . This is a quantity of interest because, from the perspective of estimation, one could still have consistency even if .
In what follows, we define , , , , and . For example, when , we have and .
For the next set of results, we let and . Due to space constraints, we prove both theorems in Appendix A.
Theorem 5**.**
Suppose is large enough so that, given and , with probability at least ,
[TABLE]
Then with probability at least ,
[TABLE]
If is independent of the training data , then, given and , with probability at least ,
[TABLE]
Remark 10**.**
The assumption (22) can be recast by saying that the distance between and is less than a constant multiple, namely , of the length of the parent subnode. (Note that by definition, .) Such an assumption is not unrealistic since if has a unique global maximum, i.e., is unique, [25, Theorem 2] shows converges weakly to zero. In fact, with similar assumptions, one can characterize the rate of convergence [43, Theorem 3.2.5]. Indeed, [3, 13] show cube root asymptotics (i.e., converges in distribution) of split points for one-level decision trees (e.g., decision stumps) using the CART sum of squares error criterion. The work of [13, Section 3.4.2] also extends these rates to multi-level decision trees, which is particularly relevant to our setting.
In practice, , , and all depend on the data . We now state a refinement of Theorem 5 that allows for data-dependent splits. Essentially it says that if the optimal empirical and population split points are sufficiently close to each other and the fraction of data points contained in the parent node is at least , then the fraction of data points contained in each daughter node is at least , where is strictly positive and depends only on , , and . In fact, the next theorem shows that one can take with high probability. This ensures that the daughter nodes contain a sufficiently large number of training samples, so that subsequent empirical splits will be close to their infinite sample counterparts, and so on and so forth.
Theorem 6**.**
Let and suppose is large enough so that with probability at least ,
[TABLE]
Then with probability at least ,
[TABLE]
and with probability at least ,
[TABLE]
The previous theorem can be used inductively to show that if the tree is grown to a depth of , then with high probability, each terminal subnode length is at most . An obvious line of future work would be to make this statement more rigorous and use it to furnish a convergence result for ensembles of decision trees with empirical splits.
10 Extension to classification trees
Our results have focused on regression trees, although nearly identical bounds can be developed for classification trees. In the binary classification context, i.e., , the variance impurity (1) equals , which is also known as Gini impurity. Here, instead of averages, the tree outputs the majority vote at a terminal node , i.e., if and otherwise.
The infinite sample Gini impurity is
[TABLE]
The analog to the conditional partial dependence function (7) is
[TABLE]
and its mean-centered version is
[TABLE]
In agreement with (6), also has the representation
[TABLE]
Thus, adapting the proofs of Theorem 7 and Theorem 9—whose original proofs also rely on such a representation for —it can easily be shown that Theorem 1 and its consequences hold verbatim with these modified definitions of , , and . In particular, Theorem 2 holds if satisfies (17). As a concrete example (c.f., Example 1 and Example 2), consider the following logistic regression model. We give the proof in Appendix A.
Example 4**.**
Let with intercept coefficient and effects coefficients and . Suppose so that . Then,
[TABLE]
and hence
[TABLE]
Curiously, this lower bound does not depend on any of the parameters other than . The importance of decreases as we become increasingly more certain that either or , i.e., as the parameter or grows, respectively.
In the next section, we provide proofs of the main results from Section 6—Theorem 1 and Theorem 2.
11 Proofs and additional results
Throughout this section, for notational clarity and brevity, we omit dependence on the input variable index for all quantities and assume that we are splitting on a generic coordinate . We also sometimes omit dependence on , and substitute and with and , respectively.
The first theorem, from which most of the results in the paper are derived, gives a clean expression for the optimally split daughter node probabilities in terms of the partial dependence function and largest reduction in impurity.
Theorem 7**.**
Suppose Assumption 1 holds and . Then
[TABLE]
and consequently,
[TABLE]
Proof.
Recall from (6) that one can write
[TABLE]
Next, define
[TABLE]
so that
[TABLE]
An easy calculation shows that
[TABLE]
Taking the derivative of with respect to , we find that
[TABLE]
Suppose is a global maximizer of (27) (in general, it need not be unique). Then a necessary condition (first-order optimality condition) is that the derivative of is zero at . That is, from (29), satisfies
[TABLE]
If and , it follows from rearranging (30) that
[TABLE]
This expresses as a fixed point of the mapping
[TABLE]
where denotes the composition operator and denotes the quantile function of the probability measure with distribution function , i.e.,
[TABLE]
The solution to (31) is obtained by solving a simple quadratic equation of the form , which proves the identity (24). The identity for the node balancedness (25) follows immediately from Definition 2 and (24). ∎
Remark 11**.**
One can make connections between the representation in Theorem 7 and other quantities defined in the literature. For example, [25, Section 2.8] define the (empirical) edge-cut preference statistic of a split as
[TABLE]
The population version is , which according to Theorem 7, is equal to at the optimal split .
The expression in Theorem 7 reveals that the optimal split is a perturbation of the median of the conditional distribution , where the gap is governed by the largest decrease in impurity, namely, , and the mean-centered partial dependence function, namely, . At the very extreme, the reduction in weighted variance is smallest when there is no signal in the splitting direction— for and . Thus, by Theorem 7, splits along directions that contain a strong signal (as opposed to noisy or directions or directions with weak signals) tend to be further away from the parent node subedges [25]. In fact, this has been empirically observed for some time [12, Section 11.8], i.e., squared error impurity tends to favor end-cut splits—that is, splits in which the proportion of data contained in an optimally split node is close to zero or one. The perturbation from the median of is zero and hence when , or equivalently, when . This is true in the special case that the regression function is linear and the input distribution is uniform, since in this case it can be shown that . Next, we state a more general result for other regression functions.
Example 5**.**
Suppose . If is uniform, then
[TABLE]
We also have the following corollary, which expresses the -probability of any terminal node in terms of the largest decrease in impurity and the conditional partial dependence function. Its proof can be deduced from a simple induction argument and Theorem 7.
Corollary 1**.**
Consider a decision tree with splits determined by optimizing the infinite sample CART objective (5). Suppose Assumption 1 holds and for all nodes . Then the -probability of any terminal node is
[TABLE]
where the product extends over all ancestor nodes of . The value of is given in Table 2.
For purposes of obtaining useful upper and lower bounds on (and therefore also ), we see from (24) that it suffices to lower bound and upper bound . The next lemma shows that is at most the oscillation of the partial dependence function over the node.
Lemma 2**.**
Suppose Assumption 1 and . Then,
[TABLE]
Furthermore, if each first-order partial derivative of the regression function and joint density exist and are continuous, then
[TABLE]
Proof.
It can be shown that . By Theorem 7, . Thus, , which is equivalent to the first claimed inequality in (32). Next, we show that . Since
[TABLE]
by the generalized mean value theorem for definite integrals, there exists such that . Hence can also be bounded by the oscillation of the partial dependence function on since
[TABLE]
This proves the inequalities in (32).
To show (33), note that when is smooth, its oscillation is bounded by its total variation . This bound is occasionally useful and will be used in the proof of Example 1, Example 2, and Example 3. ∎
Combining Theorem 7 with Lemma 2, we have the following bound on the conditional daughter node probabilities.
Theorem 8**.**
Suppose Assumption 1 and . Then both and are between
[TABLE]
and consequently,
[TABLE]
This implies that and tend to be more extreme (i.e., closer to zero or one) if the oscillation of the partial dependence function is large. Indeed, we have seen from Example 2 that the optimal split point for a sinusoidal waveform gets closer and closer to its parent subnode edges as the periodicity increases.
Note that to obtain Theorem 7 and its consequences in Theorem 8, we only used the first-order optimality condition for . Next, we show that by additionally incorporating second-order conditions, an alternate (and sometimes better) bound can be obtained. First, we state a lemma.
Lemma 3**.**
Suppose Assumption 1 holds and each first-order partial derivative of the regression function and joint density exist and are continuous. Then,
[TABLE]
Proof.
If the first-order partial derivative of the joint density exists and is continuous, then the density of the conditional probability measure , namely , is continuously differentiable in . If, in addition, the first-order partial derivative of the regression function exists and is continuous, then by Leibniz’s integral rule, exists and is therefore well-defined. We will show that if , then
[TABLE]
The conclusion (35) then follows from the fact that since is a global maximizer. Let us now show (36). We use the expression (29) as a starting point. Since , the second derivative at is equal to
[TABLE]
Next, we compute the derivative in (37) and find that
[TABLE]
Next, recall that , so that the expression in (38) is equal to
[TABLE]
Next, we multiply (39) by so that (37) is equal to
[TABLE]
Finally, observe that by the first-order condition (29), , and by definition, . ∎
Theorem 9**.**
Suppose Assumption 1 holds, , and each first-order partial derivative of the regression function and joint density exist and are continuous. Then both and are between
[TABLE]
and consequently,
[TABLE]
Proof.
First note that , since otherwise, by (36) and the second-order optimality condition, . Using the solutions to the first-order condition (31), we have that
[TABLE]
and furthermore, by the second-order condition (35)
[TABLE]
Rearranging yields
[TABLE]
Finally, it is a simple exercise to show that if , then . ∎
Remark 12**.**
If the regression surface is linear and the distribution of is uniform, then (40) is approximately equal to . Compare this with the true value of for both daughter node conditional probabilities.
We are now in a position to give the proof of Theorem 1.
Proof of Theorem 1.
Let be the parent node of . Suppose we split along coordinate . Then is at most
[TABLE]
where the first inequality follows from Assumption 2, the penultimate inequality follows from for , and the final inequality follows from Theorem 7. By induction and using , we have \mathbb{P}\left[a_{j}(\mathbf{t})\leq X_{j}\leq b_{j}(\mathbf{t})\right]\leq\exp\big{\{}-\frac{\eta}{4}\sum_{\begin{subarray}{c}\mathbf{t}^{\prime}\supset\mathbf{t}\\ j_{\mathbf{t}^{\prime}}=j\end{subarray}}[\Delta(j,s^{*};\mathbf{t}^{\prime})+|\overline{G}_{j}(s^{*};\mathbf{t}^{\prime})|^{2}]^{-1}\Delta(j,s^{*};\mathbf{t}^{\prime})\big{\}}, which proves (14) with weights (15). The lower bound on the weights (16) is a direct consequence of Theorem 9. ∎
Remark 13**.**
Note that is not needed for Theorem 1. This is because the inequality (14) remains true even if, for some , (with the convention that ).
11.1 Alternative splitting rules
To mitigate the effect of end-cut splits, as discussed in Section 6, one can subtract a positive penalty from and instead solve
[TABLE]
Intuitively, should be large when is close to the edges and small when is far from the edges. The penalty should also be proportional to so that some influence from original objection function is retained. One natural choice of penalty that meets these criteria is
[TABLE]
Of course, in practice one would use
[TABLE]
where .
This regularization procedure is not new—[12, Section 11.8] proposed that, to avoid end-cut splits, one should instead maximize multiplied by some power of , say for . Thus, acts as a multiplicative regularizer that modulates the effect of edge-cut preference in CART. The essential challenge is to choose the regularization parameter large enough so that splits away from the parent subnode edges are encouraged, but not in such a way that the homogeneity of the node (as measured by the variance in ) is unimproved. Good values of can be determined by any number of means, including cross-validation on a hold-out set of the data.
Denote the objective function by and its maximizer by (similarly define as the empirical maximizer). In Fig. 4 (c.f., Fig. 3), we illustrate the effect of regularization in determining the optimal splits for the sinusoidal waveform encountered previously in Example 2.
By a similar argument to establishing Theorem 7, the optimal satisfies
[TABLE]
With , we have and hence . Let us now obtain a further lower bound on . To this end, note that by concavity of , we have
[TABLE]
This means that
[TABLE]
Solving this for and yields the following theorem, which is a direct analog to Theorem 7. Using this, we also give a lower bound for , the global balancedness (see Definition 3) for .
Theorem 10**.**
Suppose Assumption 1 holds and . Then both and are between
[TABLE]
and consequently
[TABLE]
It is often possible to show that and hence, by virtue of (42), (25), and (43), that , where is the global balancedness for the unpenalized criterion. The next theorem (c.f., Theorem 9) shows that this is improvable to when . Using this, it can easily be shown via a modification of the proofs of Example 1 and Example 2 that and , respectively. These quantities are larger than their counterparts using the unpenalized and hence the regularization does indeed encourage splits that are farther away from the parent subnode edges.
Theorem 11**.**
Suppose Assumption 1 holds, , and each first-order partial derivative of the regression function and joint density exist and are continuous. Let . Then both and are between
[TABLE]
and consequently,
[TABLE]
Proof.
We have the chain of inequalities
[TABLE]
The first inequality is by concavity of and the second inequality is due to the fact that . The third inequality comes from the second-order derivative condition (c.f., (36)), i.e., given , the second derivative equals
[TABLE]
Finally, combining (44) with (42) and solving for and yields both statements of the theorem. ∎
11.2 Lower bounds on the node balancedness
Of special interest is , since this provides a nontrivial bound on the distance between any optimal split to its parent subnode edges. But can we expect this to hold in most settings? It is conceivable that may become extremely small when and are arbitrarily close to each other, since after all by Theorem 7, its defining quantities— and —expressed through their ratio, both approach zero. We argue that is still controlled in this case. To see this, suppose and are extremely close to each other. Then the partial dependence function is approximately linear, i.e., for some constants and and also . Hence , with equality if is uniform and the conditional regression surface is exactly linear. The statement in Theorem 2, which we now prove, makes this intuition precise. First, we state two lemmas, but defer their proofs until Appendix A.
Lemma 4**.**
If the features of are independent and satisfy Assumption 1 and the regression function satisfies (17), then is finite and for each node , .
Lemma 5**.**
Suppose the features of are independent with marginal densities that are continuous and never vanish, and the regression function satisfies (17). If , then
[TABLE]
where
[TABLE]
is the integrated decrease in impurity of the regression function with respect to the uniform distribution.
Proof of Theorem 2.
By Theorem 7, with weights is at least . Hence we need to show that global balancedness is positive. To this end, the first step in the proof involves showing that
[TABLE]
where (by Lemma 4) and is the positive constant from Lemma 5. This can be accomplished by Lemma 5 since
[TABLE]
Next, consider an term Taylor expansion of . Then, by definition of and a Taylor expansion argument, . Thus, combining (47) with the fact that (since and is continuous by assumption) and by continuity of at , we have
[TABLE]
Finally, (41) from Theorem 9 implies (46). The assumption of finite implies that
[TABLE]
Next, let (resp. ) denote the vector of lower (resp. upper) endpoints of the subnodes of . Now, since the regression function is continuous, it follows that and are both continuous on the domain .141414This can be seen from the generalized mean value theorem for integrals. Consequently, by Berge’s Maximum Theorem [1, Theorem 17.31] the mapping is continuous on and is an upper hemicontinuous correspondence on . In particular, by Theorem 7, and hence is an upper hemicontinuous correspondence on . Next, note that by Lemma 4 and Theorem 7, on , and by (46), for all points arbitrarily close to the boundary of . Hence . ∎
Acknowledgment
The author is indebted to Min Xu, Minge Xie, Samory Kpotufe, Robert McCulloch, Andy Liaw, Richard Baumgartner, and Michael Kosorok for helpful discussion and feedback. The author would also like to thank Joowon Klusowski for her help with the proof of Lemma 5.
Appendix A Supplemental Material
In this appendix, we give proofs of Lemma 1, Theorem 4, Lemma 4, Lemma 5, Theorem 5, and Theorem 6. We also give proofs of the examples from Section 6.4, Section 9.1, and Section 10.
A Proofs of main lemmas
Proof of Lemma 1.
We proceed by induction. The case is trivial, since clearly implies . Now let be arbitrary and assume that the claim is true for all smaller values of . Let be distinct (real or complex) polynomials without constant terms and be (real or complex) polynomials with
[TABLE]
If all are zero then we are done. Otherwise (without loss of generality) . First we divide (48) by , yielding
[TABLE]
Differentiating the identity (49) gives
[TABLE]
Multiply (49) by and (A) by . Subtracting the two resultant expressions from each other yields
[TABLE]
Now we can apply the induction hypotheses, since the are distinct polynomials without constant terms. It follows that
[TABLE]
If , then (51) is impossible since and hence . Therefore for , which also implies that . ∎
Proof of Theorem 4.
We follow the proof of [37, Lemma 2] for quantile forests, but adapted to our setting. Let denote the node containing of the tree built with randomness at the step. By [37, Theorem 4.1], we will be done if we can show that in -probability, where is the terminal node of the tree containing . Since is continuous on , it is also uniformly continuous. Hence, for each , there exists such that if , then . Hence, we must show that in probability. To this end, let , , and . Let denote the coordinate selected to split along at the step of the tree. Suppose . Then there are two cases:
The next split in is performed along the coordinate and, in that case, one of the two resulting nodes has an empty intersection with . 2. 2.
The next split in is performed along a coordinate other than the and, in that case, the two resultant nodes have a non-empty intersection with .
Thus,
[TABLE]
where is the (unique) node at the step of the forest construction that contains . By induction and (25), this implies that if with -probability one. Finally, consider a partition of into hypercubes of side length with sides determined by the hyperplanes , where and . If belongs to one of the hypercubes, then . There are at most such hyperplanes and hence
[TABLE]
if with -probability one, where ranges over all hyperplanes of the form , with and . The conditions of the theorem and [37, Theorem 4.1] imply the conclusion. ∎
Proof of Lemma 4.
Let and suppose exists and is continuous for all . By Leibniz’s integral rule, we have
[TABLE]
By (52) and the generalized mean value theorem for integrals, there exists such that . By assumption that (17) holds there exists a finite integer such that for each , there exists an integer such that for all . In particular, and hence is finite. Since was arbitrary in , this implies that is nonconstant on . Finally, it is easy to show that if , then . ∎
Proof of Lemma 5.
First, note that
[TABLE]
Since a maximum is larger than an average, for any prior on with density ,
[TABLE]
In particular, we choose the uniform prior, i.e., . Next, by assumption, is positive and continuous and hence
[TABLE]
and
[TABLE]
where the convergence is uniform. Thus, we assume henceforth that is uniform. The proof for general follows similarly.
Let denote the divided difference and . Then, we can rewrite via
[TABLE]
Fix and use a Taylor expansion of about the point and continuity of at to argue that
[TABLE]
where the convergence is uniform and the limit is nonzero by assumption. Without loss of generality, assume . By uniform continuity, there exists such that if , then
[TABLE]
Using the fact that and Jensen’s inequality, the expression
[TABLE]
is at least
[TABLE]
which by Fubini’s theorem is equal to
[TABLE]
The leading terms in in the integrand of (54) are, up to signs, and . However, note that
[TABLE]
which, per (53), is at most . Furthermore, per (53), is between and . Hence, we have that . This shows that (54) is as . Hence, we may assume that is bounded in the limit infimum (45). Using this, we have that \liminf_{(a,b)\to(c,c)}\frac{\Delta(s^{*};\mathbf{t})}{\big{(}\frac{(b-a)^{R}\overline{F}^{(R)}(c;\mathbf{t})}{R!}\big{)}^{2}} is at least
[TABLE]
Next, let us evaluate the infimum (55). In fact, we will show that it is achieved at .
Towards this end, elementary calculations reveal that the expression inside the infimum (55) is equal to
[TABLE]
Define , so that the integral (56) can be written as
[TABLE]
We first catalogue some facts about .
If is even, then . 2. 2.
. 3. 3.
If is even, , and , then . 4. 4.
. 5. 5.
.
By the second fact and the representation (57), it follows that (56) is symmetric about . Thus, it can be assumed that .
Using the fifth fact, we have that the derivative of (56) with respect to is
[TABLE]
Assume without loss of generality that is even. By the first and fourth facts, is increasing in and is nonnegative. Hence (58) is at least
[TABLE]
Note that is odd about and positive when . Using this observation, we have that (59) is equal to
[TABLE]
and nonnegative by the third fact. Therefore, (56) is increasing when and hence minimized at . Thus, the infimum (55) is equal to
[TABLE]
Routine calculations also reveal that (60) is .
Finally, let us verify all five facts above. The second, fourth, and fifth facts are straightforward. The first fact holds since is the difference between a point and a chord that lies above it on the convex function . To show the third fact, note that
[TABLE]
which is negative since for and . Since has roots at and , it follows that in this regime. ∎
Remark 14**.**
The same argument also works if the uniform prior is replaced by any symmetric prior about .
Proof of Theorem 5.
Combining (21) and (22), it can be deduced via the triangle inequality that
[TABLE]
with probability at least . Next, by (74) and continuity of , we have that
[TABLE]
We use this identity to derive lower bounds on and over split points with . Now, from (61), we have that
[TABLE]
and hence . Furthermore,
[TABLE]
and hence . We deduce that if , then
[TABLE]
By the assumption that is independent of the training data and Lemma 9 in Appendix A, given , is distributed and is distributed (the features of need not be independent for this to hold). If , then given , has the same distribution as the empirical distribution function of a sample from the distribution of . Hence by the Dvoretzky-Kiefer-Wolfowitz inequality [31], it can be shown that with probability at least , uniformly over all split points ,
[TABLE]
where . In particular, for and , with probability at least
[TABLE]
[TABLE]
Since and , the quantities in (63) are interpretable as lower bounds on the fraction of data points in the optimal daughter node ( and ) that are contained in the parent node . A consequence of this analysis is that if is not too small, this fraction is non-negligible. ∎
Proof of Theorem 6.
The proof is similar to Theorem 5, although we must use a stronger concentration inequality to control an empirical process over a collection of nodes. That is, the proof is based on simultaneous control of the empirical processes
[TABLE]
where is the collection of all nodes in dimensions. To this end, define and so that . We make use of the inequality
[TABLE]
which can be deduced from the triangle inequality. We would like to obtain an upper bound on the probability that
[TABLE]
for . On an event with probability at least , it holds that and and (using (62) from the proof of Theorem 5), and hence (65) is contained in the event
[TABLE]
with probability at least , where . Using (64), this event is also contained in
[TABLE]
with probability at least . By Lemma 6, each event above has probability at most for a total probability of at most . The proof is completed by choosing . ∎
Lemma 6**.**
Let be the set of all nodes in . Let and be i.i.d. random vectors in . Then for all ,
[TABLE]
where .
Proof of Lemma 6.
This follows from [44] and the fact that the VC-dimension of is . ∎
B Proofs of example regression functions
In this subsection, we give proofs of Example 1, Example 2, and Example 3 from Section 6.4 and Example 4 from Section 10.
Proof of Example 1.
Without loss of generality, we will prove the theorem when . The objective function can be expressed as
[TABLE]
Note that is at least
[TABLE]
and the derivative of the partial dependence function at is
[TABLE]
Thus, by (41),
[TABLE]
The penultimate line above follows from the inequality for and the binomial theorem. Since is arbitrary, it follows that . ∎
Remark 15**.**
If and , note that if , then as . Since , this shows that . Although we have not given a formal proof, this calculation provides evidence that .
Proof of Example 2.
Without loss of generality, we will prove the theorem when . In this case, is equal to
[TABLE]
Note the term which is equal to the difference of and the line segment between and at , or the so-called “Jensen gap”.151515Typically, this terminology is reserved for convex or concave functions, but we nevertheless use it here. A major task of the proof is in choosing a suitable so that the Jensen gap is large. Next, note that
[TABLE]
which combines a pointwise inequality of one and the total variation inequality (33) from Lemma 2.
We break the proof into two parts, depending on whether or .
Case I:
Suppose that and that is monotone on . Consider at . Then it can be shown that
[TABLE]
Since is monotone on , it follows that its total variation is equal to and hence
[TABLE]
Combining the estimates (67) and (68) and using for and for , we conclude from (34) that
[TABLE]
Thus, . Next, suppose that is neither increasing or decreasing on . This means that for some positive integer , the point belongs to . Thus, there are choices such that
[TABLE]
Since , it follows from (34) that . Moreover, using the total variation bound in (66),
[TABLE]
where the penultimate line follows from a Taylor expansion argument. Thus, by (34), and hence .
Case II:
Next, we investigate when . In this regime, is allowed to make at least a quarter period on . This means that there exists a universal constant and such that the Jensen gap
[TABLE]
and hence . Furthermore, . Thus, by (41), . ∎
Remark 16**.**
If and , and then . Furthermore, . Together these estimates imply that . As with the case of polynomials (c.f., Remark 15), it is likely that the bound is improvable to , and we leave its proof as an open question for future investigation.
Lemma 7**.**
Suppose . Then .
Proof.
In this case,
[TABLE]
The derivative of the partial dependence function is . If or , then a lower bound on is easy to state. In this case, .
Choosing , we have from (41) that
[TABLE]
If or , then and there are two cases to consider for obtaining lower bounds on .
Case I:
Choose , where . Then,
[TABLE]
Let . Then, , and hence by (41), .
Case II:
Choose , where . Then,
[TABLE]
Choosing yields , and hence (41), . ∎
Lemma 8**.**
Suppose . Then .
Proof.
First, note that is equal to
[TABLE]
In this case,
[TABLE]
Using the total variation bound (66), one can easily show that
[TABLE]
where the last line follows from for . Using for , we conclude from (34) that
[TABLE]
This implies that . ∎
Proof of Example 4.
Consider a generic coefficient of , say . First, note that we may assume without loss of generality that (since otherwise, we can consider ). In this case, is at least
[TABLE]
and
[TABLE]
The above inequality (70) is due to the fact that is decreasing. Combining inequalities (69) and (70) with (41), we have that
[TABLE]
We now consider two cases for evaluating the integral in (71).
Case I:
In this regime,
[TABLE]
Thus, .
Case II:
Alternatively, in this regime,
[TABLE]
Thus, . In summary, . Since was arbitrary, this shows that . ∎
C Proof of example distributions
Below we list the examples from Section 9.1 and give their proofs.
- (b)
If each is i.i.d. , then and and hence we can write
[TABLE]
By concavity of the square root function, . Thus, for all and so . For the other direction, we can choose if we can show that , or equivalently, that
[TABLE]
By concavity of the square root function, the expression in ((b)) is at least . 2. (c)
If each is i.i.d. , then and and hence we can write
[TABLE]
The proof is finished by noting that for all .
D Miscellaneous lemmas
Lemma 9**.**
Suppose and are events and let and be i.i.d. random variables. If and , then .
Proof.
Let and . Note that . Then
[TABLE]
which is the mass function of . ∎
Before we state the next lemma, let us first introduce some notation. For a function , we write (resp. ) to denote the left (resp. right) side limits of at , i.e., and .
Lemma 10**.**
Let be a distribution function and its quantile function, i.e., . Then,
[TABLE]
and
[TABLE]
Furthermore, if is continuous and strictly increasing, then all inequalities are equalities.
Proof.
These are standard facts from probability theory and can be deduced from the Galois inequalities. See, for example, [34, Section 2.5.2]. ∎
The reference list from the paper itself. Each links out to its DOI / PubMed record.
- 1Aliprantis and Border [2006] Charalambos D Aliprantis and Kim C Border. Infinite Dimensional Analysis: A Hitchhiker’s Guide . Springer, 2006.
- 2Arlot and Genuer [2014] Sylvain Arlot and Robin Genuer. Analysis of purely random forests bias. ar Xiv preprint ar Xiv:1407.3939 , 2014.
- 3Banerjee and Mc Keague [2007] Moulinath Banerjee and Ian W Mc Keague. Confidence sets for split points in decision trees. The Annals of Statistics , 35(2):543–574, 2007.
- 4Bentley [1975] Jon Louis Bentley. Multidimensional binary search trees used for associative searching. Communications of the ACM , 18(9):509–517, 1975.
- 5Biau [2012] Gérard Biau. Analysis of a random forests model. Journal of Machine Learning Research , 13(Apr):1063–1095, 2012.
- 6Biau and Scornet [2016] Gérard Biau and Erwan Scornet. A random forest guided tour. Test , 25(2):197–227, 2016.
- 7Biau et al. [2008] Gérard Biau, Luc Devroye, and Gábor Lugosi. Consistency of random forests and other averaging classifiers. Journal of Machine Learning Research , 9(Sep):2015–2033, 2008.
- 8Breiman [1996] Leo Breiman. Bagging predictors. Machine learning , 24(2):123–140, 1996.