Forest Representation Learning Guided by Margin Distribution

TL;DR

This paper introduces a new margin distribution reweighting method for forest representation learning, improving generalization bounds and enhancing model performance by focusing on margin distribution optimization.

Contribution

It reformulates forest learning as an additive model, tightens the generalization gap bound, and proposes mdDF to optimize margin distribution for better representation learning.

Findings

Improved generalization bound from $rac{ ext{ln} m}{m}$ to $rac{ ext{ln} m}{m}$

Effective margin distribution optimization enhances performance

Visualizations confirm improved representation learning

Abstract

In this paper, we reformulate the forest representation learning approach as an additive model which boosts the augmented feature instead of the prediction. We substantially improve the upper bound of generalization gap from $\mathcal{O}(\sqrt\frac{\ln m}{m})$ to $\mathcal{O}(\frac{\ln m}{m})$, while $\lambda$ - the margin ratio between the margin standard deviation and the margin mean is small enough. This tighter upper bound inspires us to optimize the margin distribution ratio $\lambda$. Therefore, we design the margin distribution reweighting approach (mdDF) to achieve small ratio $\lambda$ by boosting the augmented feature. Experiments and visualizations confirm the effectiveness of the approach in terms of performance and representation learning ability. This study offers a novel understanding of the cascaded deep forest from the margin-theory perspective and further uses the mdDF approach to guide the layer-by-layer forest representation learning.

Tables

Table 1. Table 1: Description of the dataset in terms of number of training examples, number of testing examples, number of features and number of labels.

Dataset	Attribute	Instance	Feature	Class
ADULT	Categorical	48842	14	2
YEAST	Categorical	1484	8	10
LETTER	Categorical	20000	16	26
PROTEIN	Categorical	24387	357	3
HAR	Mixed	10299	561	6
SENSIT	Mixed	78823	50	3
SATIMAGE	Numerical	6435	36	6
MNIST	Numerical	70000	784	10

Table 2. Table 2: Comparison results between mdDF and the other tree-based algorithm on test accuracy with different datasets. The best accuracy on each data set is highlighted in bold type. ∙ ∙ \bullet indicates the second accuracy on each data set. The average rank is listed in the first row from the bottom.

Dataset	MLP	R.F.	XGBoost	gcForest	mdDF
ADULT	80.597	85.818	85.904	$\bullet$86.276	86.560
YEAST	59.641	61.886	59.161	$\bullet$63.004	63.340
LETTER	96.025	96.575	95.850	$\bullet$97.375	97.500
PROTEIN	68.660	68.071	$\bullet$71.741	71.590	71.757
HAR	$\bullet$94.231	92.569	93.112	94.224	94.600
SENSIT	78.957	80.133	81.874	$\bullet$82.334	82.534
SATIMAGE	91.125	91.200	90.450	$\bullet$91.700	91.750
MNIST	$\bullet$98.621	96.831	97.730	98.252	98.734
Avg. Rank	3.650	4.000	3.750	2.375	1.000

Table 3. Table 3: Comparison results between mdDF and the other mdDF structures on test accuracy with different datasets.

Dataset	${\text{mdDF}}_{\text{SF}}$	${\text{mdDF}}_{\text{ST}}$	${\text{mdDF}}_{\text{NP}}$	mdDF
ADULT	86.200	85.710	85.650	86.560
YEAST	63.000	62.780	62.556	63.340
LETTER	96.475	97.300	96.975	97.500
PROTEIN	71.681	70.291	68.509	71.757
HAR	93.926	94.290	94.060	94.600
SENSIT	82.014	80.412	80.320	82.534
SATIMAGE	91.600	91.300	90.800	91.750
MNIST	98.254	98.101	98.240	98.734

Equations

$$F(x)=\sum _{t=1}^T{\alpha }_t{h}_t\left([x,f_{t-1}(x)]\right),$$

$$h_t(x)=g_t\left([x,f_{t-1}(x)]\right)=g_t\left(\left[x,\sum _{l=1}^{t-1}{\alpha }_l{h}_l(x)\right]\right),$$

$$\underset{D}{\mathrm{Pr}}[yF(x)<0]-\underset{S}{\mathrm{Pr}}[yF(x)<r]\le \frac{\ln {\sum }_{t=1}^T{\alpha }_t\left|{\mathcal{H}}_t\right|}{r^2}\cdot \frac{\ln m}{m}+\lambda \sqrt{\frac{\ln {\sum }_{t=1}^T{\alpha }_t\left|{\mathcal{H}}_t\right|}{r^2}\cdot \frac{\ln m}{m}},$$

$$E=\bar{E}-\bar{A},$$

$$g_t=\{\begin{array}{ll}{\mathcal{A}}_{\text{rfb}}\left([x_i;y_i{]}_{i=1}^m,\mathcal{D}\right)& t=1,\\ {\mathcal{A}}_{\text{rfb}}\left([x_i,f_{t-1}(x_i);y_i{]}_{i=1}^m,{\mathcal{D}}_t\right)& t>1.\end{array}$$

$$h_t(x)=\{\begin{array}{ll}{g}_t(x)& t=1,\\ g_t\left([x,f_{t-1}(x)]\right)& t>1,\end{array}$$

$$f_t(x)=\{\begin{array}{ll}{\alpha }_t{h}_t(x)& t=1,\\ {\alpha }_t{h}_t\left([x,f_{t-1}(x)]\right)+f_{t-1}(x)& t>1,\end{array}$$

$$h_t(x)=g_t\left([x,f_{t-1}(x)]\right)=g_t\left(\left[x,\sum _{l=1}^{t-1}{\alpha }_l{h}_l(x)\right]\right).$$

$$\tilde{F}(x)=\tilde{\sigma }(F(x))=\underset{j\in \{1,2,\dots ,s\}}{\mathrm{argmax}}\left[\sum _{t=1}^T{\alpha }_t{h}_t^j(x)\right],$$

$${\mathcal{G}}_{\mathcal{F},\mathbf{N}}=\left\{\frac{1}{n}\sum _{k=1}^T\sum _{j=1}^{N_k}{g}_{k,j}|\forall (k,j)\in [T]\times [N_k],g_{k,j}\in {\mathcal{H}}_k\right\},$$

$$\begin{array}{rl}\ln \left|{\mathcal{G}}_{\mathcal{F},\mathbf{N}}\right|& \le \ln \left(\prod _{t=1}^T|{\mathcal{H}}_t{|}^{N_t}\right)\\ & =\sum _{t=1}^T\left(N_t\ln |{\mathcal{H}}_t|\right)\\ & =n\sum _{t=1}^T({\alpha }_t\ln |{\mathcal{H}}_t|)\\ & \le n\ln \sum _{t=1}^T{\alpha }_t|{\mathcal{H}}_t|\end{array}$$

$$\underset{S,{\mathcal{G}}_{\mathcal{F},n}}{\mathrm{Pr}}[yG(x)-yF(x)\ge ϵ]\le \exp \left(\frac{-n{ϵ}^2}{2-2{\mathbb{E}}_S^2[yF(x)]+4ϵ/3}\right).$$

$$\underset{S,{\mathcal{G}}_{\mathcal{F},n}}{\mathrm{Pr}}[yG(x)-yF(x)\ge ϵ]$$

$$\le \exp \left(-\frac{\lambda nϵ}{2}\right){\mathbb{E}}_{S,G_j\in {\mathcal{G}}_{\mathcal{F},n}}\left[\exp \left(\frac{\lambda }{2}\sum _{j=1}^n(y{G}_j(x)-yF(x))\right)\right]$$

$$=\exp \left(-\frac{\lambda nϵ}{2}\right)\prod _{j=1}^n{\mathbb{E}}_{S,G_j\in {\mathcal{G}}_{\mathcal{F},n}}\left[\exp \left(\frac{\lambda }{2}(y{G}_j(x)-yF(x))\right)\right]$$

$${\mathbb{E}}_{S,G_j\in {\mathcal{G}}_{\mathcal{F},n}}\left[\exp \left(\frac{\lambda }{2}(y{G}_j(x)-yF(x))\right)\right]$$

$$\le 1+{\mathbb{E}}_S[1-(yF(x){)}^2]\frac{e^{\lambda }-1-\lambda }{4}$$

$$\le \exp (1-{\mathbb{E}}_S^2[yF(x)])\frac{e^{\lambda }-1-\lambda }{4}$$

$$\underset{S,{\mathcal{G}}_{\mathcal{F},n}}{\mathrm{Pr}}[yG(x)-yF(x)\ge ϵ]\le \exp \left(\frac{n(e^{\lambda }-1-\lambda )(1-{\mathbb{E}}_S[yF(x)])}{4}-\frac{\lambda nϵ}{2}\right)$$

$$e^{\lambda }-\lambda -1=\sum _{i=2}^{\infty }\frac{{\lambda }^i}{i!}\le \frac{{\lambda }^2}{2}\sum _{m=0}^{\infty }\frac{{\lambda }^m}{3^m}=\frac{{\lambda }^2}{2(1-\lambda /3)}.$$

$$-\frac{\lambda ϵ}{2}+\frac{{\lambda }^2(1-{\mathbb{E}}_S^2[yF(x)])}{8(1-\lambda /3)}\le \frac{-{ϵ}^2}{2-2{\mathbb{E}}_S^2[yF(x)]+4ϵ/3}$$

$$\underset{D}{\mathrm{Pr}}[yF(x)<0]\le \underset{r\in (0,1]}{\inf }\left[\hat{R}+\frac{1}{m^d}+\frac{3\sqrt{\mu }}{m^{3/2}}+\frac{7\mu }{3m}+\lambda \sqrt{\frac{3\mu }{m}}\right]$$

$$\hat{R}$$

$$d$$

$$\mu$$

$$\lambda$$

$$\mathrm{Pr}[\frac{1}{m}\sum _{i=1}^m{X}_i\ge \mathbb{E}[X]+ϵ]\le \exp \left(-\frac{m{ϵ}^2}{2}\right),$$

$$\mathrm{Pr}[\frac{1}{m}\sum _{i=1}^m{X}_i\le \mathbb{E}[X]-ϵ]\le \exp \left(-\frac{m{ϵ}^2}{2}\right).$$

$$\frac{1}{m}\sum _{i=1}^m\mathbb{E}[X_i]-\frac{1}{m}\sum _{i=1}^m{X}_i\le \sqrt{\frac{2{\hat{V}}_m\ln (2/\delta )}{m}}+\frac{7\ln (2/\delta )}{3m},$$

$$\frac{1}{m}\sum _{i=1}^m\mathbb{E}[X_i]-\frac{1}{m}\sum _{i=1}^m{X}_i\ge -\sqrt{\frac{2{\hat{V}}_m\ln (2/\delta )}{m}}-\frac{7\ln (2/\delta )}{3m}.$$

$$\underset{D}{\mathrm{Pr}}[yF(x)<0]$$

$$\le \exp (-n{\beta }^2/2)+\underset{D,{\mathcal{G}}_{\mathcal{F},n}}{\mathrm{Pr}}[yG(x)<\beta ].$$

$$\underset{D}{\mathrm{Pr}}[yG(x)<\beta ]$$

$$=\underset{S}{\mathrm{Pr}}[yG(x)<\beta ]+\sqrt{\frac{2}{m}{\hat{V}}_m\sum _{i=1}^T{N}_t\ln \left(\frac{2|{\mathcal{H}}_t|}{\delta }\right)}+\frac{7}{3m}\sum _{i=1}^T{N}_t\ln \left(\frac{2|{\mathcal{H}}_t|}{\delta }\right)$$

$$\le \underset{S}{\mathrm{Pr}}[yG(x)<\beta ]+\sqrt{\frac{2n}{m}{\hat{V}}_m\sum _{i=1}^T{\alpha }_t\ln \left(\frac{2|{\mathcal{H}}_t|}{\delta }\right)}+\frac{7n}{3m}\sum _{i=1}^T{\alpha }_t\ln \left(\frac{2|{\mathcal{H}}_t|}{\delta }\right)$$

$$\le \underset{S}{\mathrm{Pr}}[yG(x)<\beta ]+\sqrt{\frac{2n}{m}{\hat{V}}_m\ln \left(\frac{2{\sum }_{i=1}^T{\alpha }_t|{\mathcal{H}}_t|}{\delta }\right)}+\frac{7n}{3m}\ln \left(\frac{2{\sum }_{i=1}^T{\alpha }_t|{\mathcal{H}}_t|}{\delta }\right)$$

$${\hat{V}}_m=\sum _{i\ne j}\frac{(\mathbb{I}[y_{i}G(x_i)<\beta ]-\mathbb{I}[y_{j}G(x_j)<\beta ]{)}^2}{2m(m-1)},$$

Full text

\cortext

[cor1]Corresponding author. Email: zhouzh@nju.edu.cn

1 Introduction

In recent years, deep neural networks have achieved excellent performance in many application scenarios such as face recognition and automatic speech recognition (ASR) (LeCun et al., 2015). However, deep neural networks are difficult to be interpreted. This defect severely restricts the development of deep learning in a few application scenarios, where the model’s interpretability is needed. Moreover, the deep neural networks are very data-hungry due to the large complexity of the models, which means that the model’s performance can decrease significantly when the size of the training data decreases (Elsayed et al., 2018; Lv et al., 2018).

In many real tasks, due to the high cost of data collection and labeling, the amount of labeled training data may be insufficient to train a deep neural network. In such a situation, traditional learning methods such as random forest (R.F.) (Breiman, 2001), gradient boosting decision tree (GBDT) (Friedman, 2001; Chen and Guestrin, 2016), support-vector machines (SVMs) (Cortes and Vapnik, 1995), etc., are still good choices. By realizing that the essence of deep learning lies in the layer-by-layer processing, in-model feature transformation, and sufficient model complexity (Zhou and Feng, 2018), recently Zhou and Feng (2017) proposed the deep forest model and the gcForest algorithm to achieve forest representation learning. It can achieve excellent performance on a broad range of tasks, and can even perform well on small or middle-scale of data. Later on, a more efficient improvement was presented (Pang et al., 2018), and it shows that forest is able to do auto-encoder which thought to be a specialty of neural networks (Feng and Zhou, 2018). The tree-based multi-layer model can even do distributed representation learning which was thought to be a special feature of neural networks (Feng et al., 2018). Utkin and Ryabinin (2018) proposed a Siamese deep forest as an alternative to the Siamese neural networks to solve the metric learning tasks.

Though deep forest has achieved great success, its theoretical exploration is less developed. The layer-by-layer representation learning is important for the cascaded deep forest, however, the cascade structure in deep forest models does not have a sound interpretation. We attempt to explain the benefits of the cascaded deep forest in the view of boosted representations.

1.1 Our results

In Section 2, we reformulate the cascade deep forest as an additive model (strong classifier) optimizing the margin distribution:

[TABLE]

where $\alpha_{t}$ is a scalar determined by $\ell_{\text{md}}$ - the margin distribution loss function reweighting the training samples. The input of forest block $h_{t}$ are the raw feature $x$ and the augmented feature $f_{t-1}=\sum_{l=1}^{t-1}\alpha_{l}h_{l}$:

[TABLE]

which is defined by such a recursion form. Unlike all the weak classifiers of traditional boosting are chosen from the same hypotheses set $\mathcal{H}$, the layer-$t$ hypotheses set in the cascade deep forest contains that of the previous layer, i.e., $\mathcal{H}_{t-1}\subset\mathcal{H}_{t},\forall t\geq 2$, due to $h_{t}$ is recursive. We name such a cascaded representation learning algorithm margin distribution deep forest (mdDF).

In Section 3, we give a new upper bound on the generalization error of such an additive model:

[TABLE]

where $m$ is the size of training set, $r$ is a margin parameter, $\lambda=\sqrt{\frac{\mathrm{Var}[yF(x)]}{\mathbb{E}_{S}^{2}[yF(x)]}}$ is a ratio between the margin standard deviation and the expected margin, $yF(x)$ denotes the margin of the samples.

Margin distribution. We prove that the generalization error can be bounded by $\mathcal{O}(\frac{\ln m}{m}+\lambda\sqrt{\frac{\ln m}{m}})$. When the margin distribution ratio $\lambda$ is small enough, our bound will be dominated by the higher order term $\mathcal{O}(\frac{\ln m}{m})$. This bound is tighter than previous bounds proved by Rademacher’s complexity $\mathcal{O}(\sqrt{\frac{\ln m}{m}})$ (Cortes et al., 2014). This result inspires us to optimize the margin distribution by minimizing the ratio $\lambda$. Therefore, we utilize an appropriate margin distribution loss function $\ell_{\text{md}}$ to optimize the first- and second-order statistics of margin.

Mixture coefficients. As for the overfitting risk of such a deep model, our bound inherits the conclusion in Cortes et al. (2014). The cardinality of hypotheses set $\mathcal{H}$ is controlled by the mixture coefficients $\alpha_{t}$s in equation 1. The hypotheses-set term $\sum_{t=1}^{T}\alpha_{t}|\mathcal{H}_{t}|$ in our bound implies that, while some hypothesis sets used for learning could have a large complexity, this may not be detrimental to generalization if the corresponding total mixture weight is relatively small. In other words, the coefficients $\alpha_{t}$s need to minimize the expected margin distribution loss $\mathbb{E}_{x\sim S}[\ell_{md}\left(\sum_{l=1}^{t}\alpha_{l}\gamma_{l}(x)\right)]$, which implies the generalization ability of the $t$ layer cascaded deep forest.

Extensive experiments validate that mdDF can effectively improve the performance on classification tasks, especially for categorical and mixed modeling tasks. More intuitively, the visualizations of the learned features in Figure 7 and Figure 9 show great in-model feature transformation of the mdDF algorithm. The mdDF not only inherits all merits from the cascaded deep forest but also boosts the learned features over layers in cascade forest structure.

1.2 Additional related work

The gcForest (Zhou and Feng, 2018) is constructed by multi-grained scanning operation and cascade forest structure. The multi-grained scanning operation aims to dealing with the raw data which holds spatial or sequential relationships. The cascade forest structure aims to achieving in-model feature transformation, i.e., layer-by-layer representation learning. It can be viewed as an ensemble approach that utilizes almost all categories of strategies for diversity enhancement, e.g., input feature manipulation and output representation manipulation (Zhou, 2012).

Krogh and Vedelsby (1995) have given a theoretical equation derived from error-ambiguity decomposition:

[TABLE]

where $E$ denotes the error of an ensemble, $\bar{E}$ denotes the average error of individual classifiers in the ensemble, and $\bar{A}$ denotes the average ambiguity, later called diversity, among the individual classifiers. This offers general guidance for ensemble construction, however, it cannot be taken as an objective function for optimization, because the ambiguity is mathematically defined in the derivation and cannot be operated directly.

In this paper, we will use the margin distribution theory to analyze the cascade structure in deep forest and guide its layer-by-layer representation learning. The margin theory was first used to explain the generalization of the Adaboost algorithm (Schapire et al., 1998; Breiman, 1999). Then a sequence of research (Reyzin and Schapire, 2006; Wang et al., 2011; Gao and Zhou, 2013) tries to prove the relationship between the generalization gap and the empirical margin distribution for boosting algorithm. Cortes et al. (2017) propose a deep boosting algorithm which boosts the accuracy of variant depth decision trees, and Cortes et al. (2017); Huang et al. (2018) offer a Rademacher bcomplexity analysis to deep neural networks. However, these theoretical results depend on the Rademacher complexity rather than the margin distribution. Since the Rademacher complexity of the forest module cannot be explicitly formulized, it cannot be taken as an objective function for optimization.

2 Cascaded Deep Forest

As shown in Figure 1, the cascaded deep forest is composed of stacked entities referred to as forest block $g_{t}$s. Each forest block consists of several forest modules, which are commonly RF (random forest) (Breiman, 2001) and CRF (Completely-random forest) (Zhou and Feng, 2017). Cascade structure transmits the samples’ representation layer-by-layer by concatenating the augmented feature $f$ onto the origin input feature $x$. In fact, we can name this operation “preconc" (prediction concatenation), because the augmented feature is the prediction scores of forests in each layer. It is worth noting that “preconc" is completely different from the stacking operation (Wolpert, 1992; Breiman, 1996) in traditional ensemble learning. The second-level learners in stacking act on the prediction space composed of different base learners and the information of origin input feature space is ignored. Using the stacking operation with more than two layers would suffer seriously from overfitting in the experiment, and cannot enable a deep model by itself. The cascade structure is the key to success of forest representation learning, however, there has been no explicit explanation for this layer-by-layer process yet.

Firstly we reformulate the cascaded deep forest as an additive model mathematically in this section. We consider training and test samples generated i.i.d. from distribution $\mathcal{D}$ over $\mathcal{X}\times\mathcal{Y}$, where $\mathcal{X}\in\mathbb{R}^{n}$ is the input space and $\mathcal{Y}\in\{1,2,\dots,s\}$ is the label space. We denote by $S$ a training set of $m$ samples drawn according to $\mathcal{D}^{m}$. $\mathcal{H}_{1},\mathcal{H}_{2},\dots,\mathcal{H}_{T}$ denote $T$ families ordered by increasing complexity, i.e., $\mathcal{H}_{1}\subset\mathcal{H}_{2}\dots\subset\mathcal{H}_{T}$.

A cascaded deep forest algorithm can be formalized as follows. We use a quadruple form $({\bm{g}},{\bm{f}},\bm{\mathcal{D}},{\bm{h}})$ where

$\bullet$  ${\bm{g}}=(g_{1},g_{2},\dots,g_{T})$, where $g_{t}$ denotes the function computed by the $t$-th forest block which is defined by equation 5;

$\bullet$  ${\bm{h}}=(h_{1},h_{2},\dots,h_{T})$, where $h_{t}$ denotes the $t$ layer cascaded forest defined by equation 6, and $h_{t}$ drawn from the hypothesis set $\mathcal{H}_{t}$;

$\bullet$  ${\bm{f}}=(f_{1},f_{2},\dots,f_{T})$, where $f_{t}$ denotes the augmented feature in layer $t$, which is defined by equation 7;

$\bullet$  $\bm{\mathcal{D}}=(\mathcal{D}_{1},\mathcal{D}_{2},\dots,\mathcal{D}_{T})$, where $\mathcal{D}_{t}$ is the updated sample distribution in layer $t$.

$g_{t}$ is the $t$-level weak module returned by the random forest block algorithm 1. It is learned from the raw training sample $S=\{(x_{1},y_{1}),\dots,(x_{m},y_{m})\}$ and the augmented feature from the previous layer $f_{t-1}(x_{i}),i\in[m]$ and the reweighting distribution $\mathcal{D}_{t}$:

[TABLE]

With these weak modules, we can define the $t$ layer cascaded deep forest as:

[TABLE]

The augmented feature $f_{t}:\mathcal{X}\rightarrow\mathcal{C}$ is defined as follows:

[TABLE]

where $\alpha_{t}$ and $\mathcal{D}_{t}$ is need to be optimized and updated.

Here, we find that the $t$ layer cascaded deep forest is defined as a recursion form:

[TABLE]

Unlike all the weak classifiers of traditional boosting are chosen from the same hypotheses set $\mathcal{H}$, the layer-$t$ hypotheses set in the cascade deep forest contains that of the previous layer, similar to the hypotheses sets of the deep neural networks (DNNs) in different depth, i.e., $\mathcal{H}_{t-1}\subset\mathcal{H}_{t},\forall t\geq 2$.

The entire cascaded model $\tilde{F}:\mathcal{X}\rightarrow\mathcal{Y}$ is defined as follows:

[TABLE]

where $F(x)$ is the final prediction vector of cascaded deep forest for classification and $\tilde{\sigma}$ denotes a map from average prediction score vector to a label.

Note. Here we generalize the formula of cascaded deep forest as an additive model. In fact, the original version (Zhou and Feng, 2017) sets the data distribution invariable $\mathcal{D}_{t}\equiv\mathcal{D}$ and the augmented feature non-additive $f_{t}=h_{t}([x,f_{t-1}(x)])$, even the final prediction vector is the direct output $F(x)=h_{T}(x)$. Through the generalization analysis in next section, we will explain why we need to optimize $\alpha_{t}$ and update $\mathcal{D}_{t}$.

3 Generalization Analysis

In this section, we analyze the generalization error to understand the complexity of the cascaded deep forest model. For simplicity, we consider the binary classification task. We define the strong classifier as $F(x)=\sum_{t=1}^{T}\alpha_{t}h_{t}$, i.e., the cascaded deep forest reformulated as an additive model. Now we define the margin for sample $(x,y)$ as $yF(x)\in[-1,1]$, which implies the confidence of prediction. We assume that the hypotheses set $\mathcal{H}$ of base classifiers $\{h_{1},h_{2},\dots,h_{T}\}$ can be decomposed as the union of $T$ families $\mathcal{H}_{1},\mathcal{H}_{2},\dots,\mathcal{H}_{T}$ ordered by increasing complexity, where $\forall t\geq 1,\mathcal{H}_{t}\subset\mathcal{H}_{t+1}\ \text{and}\ h_{t}\in\mathcal{H}_{t}$. Remarkably, the complexity term of these bounds admits an explicit dependency in terms of the mixture coefficients defining the ensembles. Thus, the ensemble family we consider is $\mathcal{F}=\operatorname{conv}\left(\bigcup_{t=1}^{T}\mathcal{H}_{t}\right)$, that is the family of functions $F(x)$ of the form $F(x)=\sum_{t=1}^{T}\alpha_{t}h_{t}(x)$, where $\bm{\alpha}=\left(\alpha_{1},\dots,\alpha_{T}\right)$ is in the simplex $\Delta$.

For a fixed $\mathbf{g}=(g_{1},\dots,g_{T})$, any $\bm{\alpha}\in\Delta$ defines a distribution over $\{g_{1},\dots,g_{T}\}$. Sampling from $\{g_{1},\dots,g_{T}\}$ according to $\bm{\alpha}$ and averaging leads to functions $G=\frac{1}{n}\sum_{i=1}^{T}n_{t}g_{t}$ for some $\mathbf{n}=(n_{1},\dots,n_{T})$, with $\sum_{t=1}^{T}n_{t}=n$, and $g_{t}\in\mathcal{H}_{t}$. For any $\mathbf{N}=(N_{1},\dots,N_{T})$ with $|\mathbf{N}|=n$, we consider the family of functions

[TABLE]

and the union of all such families $\mathcal{G}_{\mathcal{F},n}=\bigcup_{|\mathbf{N}=n|}\mathcal{G}_{\mathcal{F},\mathbf{N}}$. For a fixed $\mathbf{N}$, the size of $\mathcal{G}_{\mathcal{F},\mathbf{N}}$ can be bounded as follows:

[TABLE]

Our margin distribution theory is based on a new Bernstein-type bound as follows:

Lemma 1

For $F=\sum_{t=1}^{T}\alpha_{t}g_{t}\in\mathcal{F}$ and $G\in\mathcal{G}_{\mathcal{F},n}$, we have

[TABLE]

**Proof. **For $\lambda>0$, according to the Markov’s inequality, we have

[TABLE]

where the last inequality holds from the independent of $G_{i}$. Notice that $|yG_{j}(x)-yF(x)|\leq 2$ (the margin is bounded: $yF(x)\in[-1,1]$), using Taylor’s expansion, we get

[TABLE]

where the last inequality holds from Jensen’s inequality and $1+x\leq e^{x}$. Therefore, we have

[TABLE]

If $0<\lambda<3$, then we could use Taylor’s expansion again to have

[TABLE]

Now by picking $\lambda=\frac{\epsilon}{1/2-\mathbb{E}_{S}^{2}[yF(x)]/2+\epsilon/3}$, we have

[TABLE]

By Combining the equation 19 and 21 together, we complete the proof. ∎

Since the gap between the margin of strong classifier $yF(x)$ and the margin of classifiers in the union family $\mathcal{G}_{\mathcal{F},\mathbf{N}}$ is bounded by the margin mean, we can further obtain a margin distribution theorem as follows:

Theorem 1

Let $\mathcal{D}$ be a distribution over $\mathcal{X}\times\mathcal{Y}$ and $S$ be a sample of $m$ examples chosen independently at random according to $\mathcal{D}$. With probability at least $1-\delta$, for $r>0$, the strong classifier $F(x)$ (depth-$T$ mdDF) satisfies that

[TABLE]

where

[TABLE]

Proof.

Lemma 2

[Chernoff bound (Chernoff et al., 1952)] Let $X,X_{1},\dots,X_{m}$ be $m+1$ i.i.d. random variables with $X\in[0,1]$. Then, for any $\epsilon>0$, we have

[TABLE]

Lemma 3

[Gao and Zhou (2013)] For independent random variables $X_{1},X_{2},\dots,X_{m}(m\geq 5)$ with values in $[0,1]$, and for $\delta\in(0,1)$, with probability at least $1-\delta$ we have

[TABLE]

where $\hat{V}_{m}=\sum_{i\neq j}(X_{i}-X_{j})^{2}/2m(m-1)$

For $F=\sum_{t=1}^{T}\alpha_{t}g_{t}\in\mathcal{F}$ and $G\in\mathcal{G}_{\mathcal{F},n}$, we have $\mathbb{E}_{G\in\mathcal{G}_{\mathcal{F},n}}[G]=F$. For $\beta>0$, the Chernoff’s bound in Lemma 2 gives

[TABLE]

Recall that $|\mathcal{G}_{\mathcal{F},N}|\leq\prod_{t=1}^{T}|\mathcal{H}_{t}|^{N_{t}}$ for a fixed $N$. Therefore, for any $\delta_{n}>0$, combining the union bound with Lemma 3 guarantees that with probability at least $1-\delta_{n}$ over sample $S$, for any $G\in\mathcal{G}_{\mathcal{F},N}$ and $\beta>0$

[TABLE]

where

[TABLE]

The inequality 30 is a large probability bound when $n$ is large enough and inequality 31 is according to the Jensen’s Inequality. Since there are $T$ at most $T^{n}$ possible $T$-tuples $N$ with $|N|=n$, by the union bound, for any $\delta>0$, with probability at least $1-\delta$, for all $G\in\mathcal{G}_{\mathcal{F},n}$ and $\beta>0$:

[TABLE]

Meantime, we can rewrite $\hat{V}_{m}$

[TABLE]

For any $\theta_{1},\theta_{2}>0$, we utilize Chernoff’s bound in Lemma 3 to get:

[TABLE]

where $\mathrm{Var}[yF(x)]=\mathbb{E}_{S}[(yF(x))^{2}]-\mathbb{E}_{S}^{2}[yF(x)]$ is the variance of the margins.

From Lemma 1, we obtain that

[TABLE]

Let $\theta_{1}=r/6$, $\beta=5r/6$ and $n=\ln m/r^{2}$, then we combine the equation 27,28,43 and 44, the proof is completed.

∎

Remark 1. From Theorem 1, we know that the gap between the generalization error and empirical margin loss is generally bounded by the forests complexity $\mathcal{O}\left(\sqrt{\frac{\ln m\ln(\sum_{t=1}^{T}\alpha_{t}|\mathcal{H}_{t}|)}{mr^{2}}}\right)$, which is controlled by the ratio between the margin standard deviation and the margin mean $\lambda=\sqrt{\frac{\hat{V}_{m}[yF(x)]}{\mathbb{E}_{S}^{2}[yF(x)]}}$. This ratio implies that the smaller margin mean and larger margin variance can reduce the complexity of models properly, which is crucial to alleviate the overfitting problem. When the margin distribution is good enough (margin mean is large and margin variance is small), $\mathcal{O}\left(\frac{\ln m}{m}\right)$ will dominate the order of the sample complexity. This is tighter than the previous theoretical work about deep boosting (Cortes et al., 2014, 2017; Huang et al., 2018) $\mathcal{O}(\sqrt{\frac{\ln m}{m}})$.

Remark 2. Moreover, this novel bound inherits the property of previous bound (Cortes et al., 2014), the hypotheses term $\ln\sum_{t=1}^{T}\alpha_{t}|\mathcal{H}_{t}|$ admits an explicit dependency on the mixture coefficients $\alpha_{t}$s. It implies that, while some hypothesis sets used for learning could have a large complexity, this may not be detrimental to generalization if the corresponding total mixture weight is relatively small. This property also offers a potential to obtain a good generalization result through optimizing the $\alpha_{t}$s.

It is worth noting that the analysis here only considers the cascade structure in deep forest. Due to the simplification of the model, we do not analyze the details about the "preconc" operation and the influence of adopting a different type of forests, though these two operations play an important role in practice. The advantages of these operations are evaluated empirically in Section 5.

4 Margin Distribution Optimization

The generalization theory shows the importance of optimizing the margin distribution ratio $\lambda$ and the mixture coefficients $\alpha_{t}$s. Since we reformulate the cascaded deep forest as a additive model, we utilize the reweighting approach to minimize the expected margin distribution loss

[TABLE]

where the margin distribution loss function $\ell_{\text{md}}$ is designed to utilize the first- and second-order statistics of margin distribution. The reweighting approach helps the model to boosts the augmented feature in deeper layer, i.e., $h_{t}$ focus on dealing with the samples which have the large loss in the previous layers. The choice of scalar $\alpha_{t}$ is determined by minimizing the expected loss for $t$-layer model.

4.1 Algorithm for mdDF approach.

We denote by $C=\mathbb{R}^{s}$ a prediction score space, where $s$ is the number of classes. When any sample $(x,y)\in\mathcal{D}$ passes through the cascaded deep forest model, it will get an average prediction vector in each layer: $h_{t}(x)=\left[h_{t}^{1}(x),h_{t}^{2}(x)\dots,h_{t}^{s}(x)\right]\in C$. According to Crammer and Singer (2001), we can define the sample’s margin $\gamma_{t}(x)$ for multi-class classification as:

[TABLE]

that is, the prediction’s confidence-rate. For example, in the 3-class problem, as shown in Figure 2, the average prediction score is $[0.2,0.3,0.5]$, and the margin is calculated as $0.5-0.3=0.2$.

The initial sample weights are $[1/m,1/m,\dots,1/m]$, and we update the $i$-th sample weight by:

[TABLE]

The margin distribution loss function $\ell_{md}(\cdot)$ is defined as follows:

[TABLE]

where hyper-parameter $\gamma$ is a parameter as the margin mean and $\mu$ is a parameter to trade off two different kinds of deviation (keeping the balance on both sides of the margin mean). In practice, we generally choose these two hyper-parameters from the finite sets $\gamma\in\{0.7,0.75,0.8,0.85,0.9,0.95\}$ and $\mu\in\{0.01,0.05,0.1\}$. The algorithm utilizing this margin distribution optimization is summarized in Algorithm 2.

4.2 The intuition of margin distribution loss function.

Since Reyzin and Schapire (2006) found that the margin distribution of AdaBoost is better than that of arc-gv (Breiman, 1999) which is a boosting algorithm designed to maximize the minimum margin, Reyzin and Schapire (2006) conjectured that margin distribution is more important to get a better generalization performance than the instance with the minimum margin. Gao and Zhou (2013) prove that utilizing both the margin mean and margin variance can portray the relationship between margin and generalization performance for AdaBoost algorithm more precisely. We list the several loss functions of the algorithms based on margin theory to compare and plot them in Figure. 3:

Compared with maximize the minimum margin (SVMs), the optimal margin distribution principle (Gao and Zhou, 2013; Zhang and Zhou, 2017) conjecture that maximizing the margin mean and minimizing the margin variance is the key to achieving a better generalization performance. Figure. 4 shows that optimizing the margin distribution with first- and second-order statistics can utilize more information on training data, e.g. the covariance among the different features. Inspired by this idea, Zhang and Zhou (2017) propose the optimal margin distribution machine (ODM) which can be formulated as equation 52:

[TABLE]

where $\theta+\xi_{i}$ and $\theta+\epsilon_{i}$ are the deviation of the margin $\gamma_{h}(x_{i},y_{i})$ to the margin mean, $\mu\in(0,1]$ is a parameter to trade off two different kinds of deviation (larger or less than margin mean). $\theta\in[0,1)$ is a parameter of the zero-loss band, which can control the number of support vectors, i.e., the sparsity of the solution, and $(1-\theta)^{2}$ in the denominator is to scale the second term to be a surrogate loss for 0-1 loss. Similar to support vector machines (SVMs), we can give ODM an intuitive illustration in Figure. 5. Similar to formulating support vector machines as a combination of the hinge loss and the regularization term, we can use margin distribution loss function $\ell_{\text{md}}$ defined in equation 51 and a regularization term to represent the ODM. Our simplified version margin distribution loss function (48) is similar to that of the ODM. Our forest representation learning approach requires as many samples as possible to train the model and generate the augmented features. Therefore, we remove the parameter $\theta$ which can control the number of support vectors. Our loss function is to optimize the margin distribution to minimize the margin distribution ratio $\lambda$, referring to Remark 2 in Section 3.

5 Experiments

In this section, we provide experiments and visualizations confirmed the effectiveness of the model in terms of performance and representation learning ability. Furthermore, we conjecture that numeric modeling tasks such as image/audio data are very suitable for DNNs because their operations, such as convolution, fit well with numeric signal modeling. The deep forest model was not developed to replace DNNs for such tasks; instead, it offers an alternative when DNNs are not superior. There are plenty of tasks, especially categorical/symbolic or mixed modeling tasks, where deep forest may be found useful.

5.1 Datasets

We choose seven classification benchmark datasets with a different scale. Table 1 presents the basic statistics of these datasets. The datasets vary in size: from 1484 up to 78823 instances, from 8 up to 784 features, and from 2 up to 26 classes. From the literature, these datasets come pre-divided into training and testing sets, therefore in our experiments, we use them in their original format. PROTEIN, SENSIT, and SATIMAGE datasets are obtained from LIBSVM data sets (Chang and Lin, 2011), except for MNIST (LeCun et al., 1998) dataset, others come from the UCI Machine Learning Repository (Dheeru and Karra Taniskidou, 2017). Based on the attribute characteristics of the dataset, we classify the datasets into three categories: categorical, numerical, and mixed modeling tasks. As shown above, we are expecting a better performance of the deep forest model on categorical and mixed modeling tasks.

5.2 Configuration and Tasks

In mdDF, we take two random forests and two completely-random forests in each layer, and each forest contains 100 trees, whose maximum depth of tree in random forest growing with the layer $t$, i.e., $d_{max}^{(t)}\in\{2t+2,4t+4,8t+8,16t+16\}$. To reduce the risk of overfitting, feature representation vector learned by each forest is generated by $k$-fold cross validation. In detail, each instance will be used as training data for $k-1$ times, produce the final class vector as augmented features for the resulting in $k-1$ class vectors, which are then averaged to next layer. (See algorithm 1)

We compare mdDF with other four common used algorithms on different datasets: multilayer perceptron (MLP), Random Forest (R.F.) (Breiman, 2001), XGBoost (Chen and Guestrin, 2016) and gcForest (Zhou and Feng, 2017). Here, we set the same number of forests as mdDF in each layer of gcForest. For random forest, we set $400\times k$ trees; and for XGBoost, we also take $400\times k$ trees. Especially, we compare them in the small sample learning task to show their resistance ability to overfitting problem.

For the multilayer perceptron (MLP) configurations, we use ReLU for activation function, cross-entropy for loss function, adadelta for optimization, no dropout for hidden layers according to the scale of training data. The network structure hyper-parameters, however, could not be fixed across tasks. Therefore, for MLP, we examine a variety of architectures on validation set, and pick the one with the best performance, then re-train the whole network on training set and report the test accuracy. The examined architectures are listed as follows: (1) input-1024-512-output; (2) input-16-8-8-output; (3) input-70-50-output; (4) input-50-30-output; (5) input-30-20-output.

Besides, we compare our mdDF with three other mdDF structures on different datasets: (1) the mdDF using same forests (use 4 random forests), i.e., $\text{mdDF}_{\text{SF}}$; (2) the mdDF using stacking (only transmit the prediction vectors to next layer), i.e., $\text{mdDF}_{\text{ST}}$; (3) the mdDF without “preconc” (only transmit the input feature vector to next layer), i.e., $\text{mdDF}_{\text{NP}}$. We hope the contrast result could demonstrate that the internal structure (different type of forests and “preconc” operation) of the deep forest is reasonable.

Since our method introduces the margin distribution optimization to the deep forest framework, we design an experiment to visualize the variation of boosted features over different layers of the model. In this way, we can know how our reweighting operation will benefit the in-model feature transformation in the deep forest model.

5.3 Results

5.3.1 Test Accuracy on Benchmark datasets

Table 2 shows that mdDF achieves a better prediction accuracy than the other methods on several datasets. Comparing with the MLP method, deep forest models almost outperform MLP on these data sets and obtain the top 2 test accuracy on categorical or mixed modeling tasks. Thus, we conjecture that deep forest model is more suitable for categorical and mixed modeling tasks. Obviously, the deep forest model gcForest and mdDF perform better than the shallow ones, and mdDF with reweighting and boosted representations outperforms gcForest across these datasets. The empirical results show that the deep model provides an improvement in performance with in-model transformation compared to the shallow models that only have invariant features. Especially under the guidance of the margin distribution reweighting, the ensemble of different depth representations brings better generalization ability for mdDF algorithm.

Table 3 shows that mdDF achieves a better prediction accuracy than the other mdDF structures on several datasets. The empirical results show that the internal structure of mdDF (different forests and “preconc” operation) is the key to achieving a better generalization performance. As we analyze in Section 3, the ensemble of learned features over different layer can alleviate the overfitting problem theoretically because coefficients $\alpha_{t}$s control the complexity. Different type of forests can enhance the diversity of deep forest model. According to the Eq. (4), higher diversity can help achieve a better performance.

5.3.2 Limited Training Samples

Recently Han et al. (2018) found the tree-based ensemble model to be more robust and stable than SVM, and neural networks, especially with small training sets. We desire that mdDF can enhance this property for tree-based model when the training data is insufficient. According to the generalization bound in Theorem 1, our algorithm can restrict the complexity of the hypothesis space suitably. To evaluate the performance of mdDF model under limited training set, we randomly chose some fraction of the training set, in particular, from 100% of the training samples to 1% on the UCI HAR dataset, and train the models accordingly.

In Figure 6, we show the test accuracies of Random Forest, XGBoost, gcForest, mdDF models trained on different fractions of the UCI HAR dataset. As shown in Figure 6, the test accuracies of all these four models increase as the fraction of training samples increases. Obviously, the mdDF model outperforms all the other models constantly across different fractions. On the UCI HAR dataset, the mdDF model outperforms Random Forest model by around 3.87%, XGBoost model by around 5.23% and gcForest model by around 2.48% on the smallest training set which contains only 1% of the whole training samples. Thus we know that the cascade structure can efficiently improve the generalization performance on limited training set with a deep model. Especially, the optimal margin distribution principle give the cascade processing a sound explanation.

5.3.3 Accuracy Curves over Layers

We use our mdDF algorithm to train and test on the UCI HAR dataset plot Figure 9 based on a similar figure in Schapire et al. (1998). It can be observed that our method achieves $100\%$ training accuracy in less than $3$ layers, but after that, the generalization accuracy keeps increasing. As we show in Theorem. 1, the margin distribution is crucial to explain why the algorithm seems resistant to overfitting problem. Especially, we can evaluate the margin distribution by calculating the ratio between the margin variance and the square of margin mean. In Figure 9, we show that the margin distribution varies with the layers, that is, the margin mean becomes larger and the margin variance become smaller, and the ratio between them is shown in Figure 9.

5.3.4 Feature Visualization

Since the performance of the mdDF model is excellent, we hope to see that the distributions of data in the learned feature space (over different layers) are consistent with the generalization ability. In this experiment, we use the t-SNE method to visualize the data distribution over different layers for training samples and test samples. Figure 7 plots the 2-dimension embedding image on UCI HAR dataset. The t-SNE (Maaten and Hinton, 2008) is a tool to visualize high-dimensional data. It converts similarities between data points to joint probabilities and tries to minimize the Kullback-Leibler divergence between the joint probabilities of the low-dimensional embedding and the high-dimensional data.

Consistently, we can find that the visualization of our model is getting better as the layer becomes deeper, the distribution of the samples which has the same label is more compact. To quantify the degree of compactness of the distribution, we perform a variance decomposition on the data in the embedding space. Compared with the ratio of the standard deviation of margin to the mean of margin in Figure 9, we can know that mdDF model attains a better distribution with a small ratio $\lambda$ while the layer becomes deeper.

6 Conclusion

Recent studies propose a few tree-based deep models to learn the representations from a broad range of tasks and achieve good performance. We extend the margin distribution theory to explain how the deep forest models learn the different representations over layers and further guide the ensemble method to obtain a better performance. We also propose the mdDF approach, which aims to get a better margin distribution, i.e., a small ratio $\lambda$. As for experiments, the results validate the superiority of our method on different datasets and show its powerful ability to learn good representations.