Online Local Boosting: improving performance in online decision trees

TL;DR

This paper introduces Online Local Boosting (OLBoost), a novel method that enhances online decision trees' predictive accuracy by boosting small regions of data without increasing memory or processing costs.

Contribution

OLBoost is a new boosting technique that improves online decision tree performance by focusing on local regions, without altering tree structures.

Findings

OLBoost significantly improves predictive accuracy of online decision trees.

Smaller trees with OLBoost can outperform larger trees.

OLBoost maintains low memory and processing costs.

Abstract

As more data are produced each day, and faster, data stream mining is growing in importance, making clear the need for algorithms able to fast process these data. Data stream mining algorithms are meant to be solutions to extract knowledge online, specially tailored from continuous data problem. Many of the current algorithms for data stream mining have high processing and memory costs. Often, the higher the predictive performance, the higher these costs. To increase predictive performance without largely increasing memory and time costs, this paper introduces a novel algorithm, named Online Local Boosting (OLBoost), which can be combined into online decision tree algorithms to improve their predictive performance without modifying the structure of the induced decision trees. For such, OLBoost applies a boosting to small separate regions of the instances space. Experimental results presented in this paper show that by using OLBoost the online learning decision tree algorithms can significantly improve their predictive performance. Additionally, it can make smaller trees perform as good or better than larger trees.

Tables

Table 1. TABLE I : Wilcoxon signed-rank test results with α = 0.05 𝛼 0.05 \alpha=0.05 : each cell shows the number of times the algorithm in the row was statistically better than the algorithm in the column, regardless the hyperparameters used.

Algorithm/Algorithm	O_SVFDT-I	O_SVFDT-II	O_VFDT	SVFDT-I	SVFDT-II	VFDT
O_SVFDT-I	–	1	1	9	7	7
O_SVFDT-II	6	–	1	8	9	7
O_VFDT	9	8	–	9	8	8
SVFDT-I	0	0	0	–	1	1
SVFDT-II	0	0	0	5	0	0
VFDT	3	2	1	7	6	–

Table 2. (a) Mean Accuracy

Algorithm/Algorithm	O_SVFDT-I	O_SVFDT-II	O_VFDT	SVFDT-I	SVFDT-II	VFDT
O_SVFDT-I	–	1	1	9	7	7
O_SVFDT-II	6	–	1	8	9	7
O_VFDT	9	8	–	9	8	8
SVFDT-I	0	0	0	–	1	1
SVFDT-II	0	0	0	5	0	0
VFDT	3	2	1	7	6	–

Table 3. (b) Memory size

Algorithm/Algorithm	O_SVFDT-I	O_SVFDT-II	O_VFDT	SVFDT-I	SVFDT-II	VFDT
O_SVFDT-I	–	10	11	0	0	6
O_SVFDT-II	0	–	11	0	0	5
O_VFDT	0	0	–	0	0	0
SVFDT-I	10	11	11	–	10	11
SVFDT-II	6	11	11	0	–	11
VFDT	2	4	11	0	0	–

Table 4. (c) Time (s)

Algorithm/Algorithm	O_SVFDT-I	O_SVFDT-II	O_VFDT	SVFDT-I	SVFDT-II	VFDT
O_SVFDT-I	–	3	3	0	0	0
O_SVFDT-II	1	–	2	0	0	0
O_VFDT	1	2	–	0	0	0
SVFDT-I	11	11	11	–	1	2
SVFDT-II	11	11	11	1	–	3
VFDT	11	11	11	2	4	–

Equations

$$\phi (x,X)=\{\begin{array}{ll}\text{True},& \text{if }x\ge \overline{X}-\sigma (X)\\ \text{False},& \text{otherwise}\end{array}$$

$$\varpi (x,X)=\{\begin{array}{ll}\text{True},& \text{if }x\ge \overline{X}+\sigma (X)\\ \text{False},& \text{otherwise}\end{array}$$

$$\lambda =(mi{n}_{\lambda }-ma{x}_{\lambda })\ast P(y)+ma{x}_{\lambda },$$

Full text

I Introduction

The amount of data gathered from different sources is growing exponentially. As these data usually come in streams, there is a need for fast, reliable, and incremental data stream mining algorithms [1, 2, 3]. These algorithms have to deal with new challenges, like concept drift (CD), time and memory constraints, and well-known problems like class imbalance and overfitting. Many efforts have been made trying to tackle these challenges. To increase predictive performance and deal with CD, ensemble learners have been employed with high success in data stream scenarios [4, 5], however at the cost of requiring more computational resources. On the other hand, although using a single classifier demands fewer resources, their predictive performance is usually lower and also are prone to failing to adapt and detect CD. Hence, a solution that increases the predictive performance without severely impacting computational resources is highly desirable [1, 6, 7, 5].

Online solutions can fail to detect important aspects of the data, e.g. instances lying in decision boundaries. This is due to the fact they only process each instance once, differently from batch solutions that exploit additional information from data distributions to improve prediction performance. The well-known Boosting ensemble approach is built upon this idea: particularly concentrate on the challenging examples. AdaBoost [8], a boosting algorithm, assigns different weights for the samples, giving more importance for the cases that are harder to predict. The same reasoning could be applied in streaming scenarios. Our idea is to focus on difficulty samples more than once when updating the learning models. Hence, reducing the needed amount of time for learning potentially complex patterns. Besides achieving better predictive performance, this approach does not bring significant impacts on the learning time and memory resources.

Taking into consideration the aspects discussed before, in this paper we present a novel algorithm for data stream classification, named Online Local Boosting (OLBoost). The OLBoost, here proposed for online decision trees (ODT), works inside each leaf to dynamically adjust the incoming instances weights towards increasing the predictive performance. It works in parallel with the online decision tree inducing algorithm and does not interfere with the decision tree induction algorithm, is used solely to predict new incoming instances. Our proposal performs local boosting in the sense that only the leaf predictors are boosted towards increasing predictive performance. This work assesses the impact of OLBoost using eleven benchmark datasets. For such, it assess the impact of OLBoost inside the Very Fast Decision Tree (VFDT) [1] and the Strict VFDT (SVFDT) [3]. Experimental results showed that, when coupled with ODT, OLBoost improves accuracy without high overheads in time and memory costs.

The paper is organised as follows: Section II presents works related to our proposal. Section III gives some background on ODT building, including a brief description of VFDT and SVFDT. Following, Section IV presents our proposal. Our experimental setup is detailed in Section V. We discuss the obtained results in Section VI and present our final considerations and some venues for future research in Section VII.

II Related Work

Many techniques have been proposed to increase the predictive performance of ODTs. They can be divided into three main groups: structural modification of the decision tree, additional prediction strategies with the same structure; and ensembles.

Modifying the structure of the decision tree is a very well-explored area. The Concept-adapting VFDT (CVFDT) algorithm, proposed by Hulten et al. [9], keeps secondary trees in memory and constantly assess these trees to check if they have a higher predictive performance than the original tree and uses a sliding window to discard outdated instances. The Hoeffding Option Tree (HOT) proposed by Pfahringer et al. [10] introduces the concept of option nodes instead of normal split nodes. An option node is essentially a split node which tests for multiple conditions at the same time. When a new instance arrives at an option node, it travels along with all children nodes where the conditions are true. Predictions are done by averaging all paths. These techniques generally revolve around building a larger decision tree or using complex algorithms to discard outdated information, which also impacts computational costs.

The second group, which uses additional prediction strategies in the leaves is where our work is situated. Gama et al. [6] first introduced the idea of functional leaves to increase the predictive performance of the VFDT. These leaves use a Naive Bayes (NB) or an Adaptive NB (ANB) algorithm to further increase predictive performance [6]. To the best of our knowledge, no further works explore other solutions which belong to this group.

Lastly, many ensembles have been proposed. Oza [4] first adapted bagging and boosting to the online scenario with the OzaBagging and OzaBoosting ensembles. Bifet et al. [11] improved OzaBagging with Leveraging Bagging (LevBag) by adding an ADWIN to monitor the error of each base learner and increasing the variability of the instances’ weights when performing bagging. Adaptive Random Forests, proposed by Gomes et al. [12], further improved LevBag with ideas from the Random Forest algorithm. Online Accuracy Updated Ensemble [13] uses a sliding window to maintain a set of weighted base learners. All these algorithms offered some increase in predictive performance. Nonetheless, this is accompanied by a great increase in computational (memory and time) costs without a sufficient increase in performance to justify [14]. The most recent studies addressing ODT predictive improvements were focused on the last group, ensembles solution. However, we believe that time and memory cost could be slightly affected when improving the predictive performance by exploring a boosting approach in the prediction strategy without modifying the ODT induction.

III Online Decision Trees

Two ODT algorithms were used in the experiments carried out in this work, VFDT and its recent variation, which focus on reducing memory costs, SVFDT.

III-A Very Fast Decision Tree

The VFDT [1] is a tree-based ML algorithm for data streams based on the Hoeffding Bound (HB) theorem. When growing a tree, the VFDT employs the HB to perform a node split. After evaluating the candidate features at a split attempt with a heuristic measure $G(.)$ (e.g., Information Gain (IG) or Gini Index (GI)), VFDT uses the HB theorem to check whether the best split candidate would remain the best if the tree received additional instances.

VFDT keeps and updates the instances class distribution in a vector at each leaf to count the number of instances from each class. Likewise, counting procedures and numerical estimators are also employed to maintain the relationship between the feature values and class distributions. By doing so, VFDT can induce a model from a single instance at a time using limited computational memory resources. Additionally, under realistic assumptions, it has the same asymptotic performance as the induction of a decision tree by a standard batch algorithm [7].

Finally, VFDT has a hyperparameter $\tau$ to support tree growth when features have similar $G(.)$ values; uses an Adaptive Naive Bayes (ANB) [6] at leaves to increase predictive performance; and uses a $GP$ hyperparameter that defines the amount of instances needed by each leaf between split attempts.

III-B Strict Very Fast Decision Tree

The SVFDT algorithm [3] modifies the VFDT to create smaller decision trees while maintaining predictive performance. SVFDT has two different versions, SVFDT-I and SVFDT-II, both following the assumptions that:

A leaf node should only split if there is a minimum uncertainty of class assumption associated with the instances, according to previous and current statistics (i.e., a high entropy). 2. 2.

All leaf nodes should observe a similar number of instances to be turned into split nodes. 3. 3.

The feature used for splitting should have a minimum relevance according to previous statistics (i.e., a high IG).

The SVFDT applies additional rules to hold tree growth using the following $\varphi$ function:

[TABLE]

where $X$ is a set of observed values, $\overline{X}$ is their mean, $\sigma(X)$ is their standard deviation, and $x$ is a new observation.

First, consider that statistics computed at the time a leaf satisfy the splits conditions of the VFDT (according to the $HB$ or $\tau$) are marked with a satisfiedVFDT. For example, the $i$-th that that this occurred, the entropy of that leaf would be marked as $H_{\text{satisfiedVFDT}_{i}}$, the $G(.)$ value of the best feature would be $G_{\text{satisfiedVFDT}_{i}}$, and the number of instances seems $n_{\text{satisfiedVFDT}_{i}}$.

The SVFDT splits a leaf when it satisfies all the conditions imposed by the VFDT and four additional constraints applied to each leaf $l$:

$\varphi(H_{l},\{H_{l_{0}},...,H_{l_{L}}\})$, 2. 2.

$\varphi(H_{l},\{H_{\text{satisfiedVFDT}_{0}},...,H_{\text{satisfiedVFDT}_{S}}\})$, 3. 3.

$\varphi(G_{l},\{G_{\text{satisfiedVFDT}_{0}},...,G_{\text{satisfiedVFDT}_{S}}\})$, 4. 4.

$n\textsubscript{{l}}\geq\overline{\{n_{\text{satisfiedVFDT}_{0}},...,n_{\text{satisfiedVFDT}_{S}}\}}$,

where $L$ is the total number of leaves and $S$ is the total number of split attempts that satisfied the VFDT constraints.

The SVFDT-II uses additional skipping mechanisms to speed-up growth when class uncertainty is too high or this uncertainty is largely reduced. It employs the following $\varpi$ function:

[TABLE]

To check if either

$\varpi(H_{l},\{H_{\text{satisfiedVFDT}_{0}},...,H_{\text{satisfiedVFDT}_{S}}\})$ or 2. 2.

$\varpi(G_{l},\{G_{\text{satisfiedVFDT}_{0}},...,G_{\text{satisfiedVFDT}_{S}}\})$

hold true, and then ignoring all the other $\varphi$ constraints.

IV Online Local Boosting

The Online Local Boosting (OLBoost) is a simple algorithm designed to increase predictive performance, primarily when working coupled with ODTs. It is based on the assumption that to increase predictive performance, instances being wrongly classified are required to be used more times when inducing a model. On the other hand, instances that are easily classified can even be dis-considered during the training phase. However, the application of this procedure may introduce some pitfalls (e.g., overfitting), since it changes the original data stream distribution. To deal with this pitfalls, OLBoost works independently from tree growth, empirically reducing this problem. More specifically, OLBoost works as the predictor of each leaf in a tree. However, it must be observed that the OLBoost can use as its core either a most common (MC) prediction, NB or ANB. Consequently, according to the strategy used different statistics about the instances need to be stored. For MC, a simple class distribution is sufficient, while for NB or ANB, it needs additional statistics about the nominal and numerical feature distributions. These additional statistics are computed in the same way as a simple VFDT using as leaf predictor the ANB.

Fig. 1 describes how OLBoost works. Since we evaluate in this work two versions of OLBoost when coupled with VFDT and SVFDT, these couplings are illustrated in the figure. Both training and prediction phases are presented in the figure. Nonetheless, the presented scheme could be easily adapted to other ODT algorithms.

The instances of an (unbounded) data stream are presented one at a time to the online learning algorithm. Each instance is defined as $i=(\mathbf{x},y)$, where $\mathbf{x}$ is the feature vector and $y$ the real class. First, an instance $i$ is presented to the algorithm and sorted to a leaf in the ODT. This process corresponds to traversing the tree according to its split nodes until $i$ arrives at a leaf. Then, OLBoost tries to predict this instance class, outputting a probabilities vector which is stored. When using NB or ANB as the core of OLBoost, the probabilities outputted by these algorithms are not true probabilistic representations (between 0 and 1 and with a sum of 1). To deal with this, we tested both softmax or simply diving each probability by the sum of all probabilities and found out that the latter yielded better results.

Afterwards, the boosting procedure is performed in a similar manner as described in [4], by sampling weights from a Poisson distribution. It works by computing a $\lambda$ variable that is then used to select the instance weight $w$. Note that $w$ is equivalent to the number of times that an instance is used to update the OLBoost. Our proposal computes the instance weights by linearly combining a range of allowed Poisson distribution parameters and the probability outputted by the ODT model. In this sense, to compute $\lambda$ the following equation is used:

[TABLE]

where $min_{\lambda}$ and $max_{\lambda}$ are hyperparameters which delimit the minimum and maximum possible values for $\lambda$ and $P(y)$ corresponds to the probability estimated by the leaf of $i$ being from its real class. Then, $w$ is drawn by sampling from the Poisson($\lambda$) distribution. Note that $\lambda$ and $P(y)$ are inversely proportional, i.e., as the prediction becomes more accurate, $\lambda$ decreases, which in turn statistically reduces the chance of $w$ being a larger value. Lastly, the statistics used by the predictor inside the OLBoost (MC, NB or ANB) are updated $w$ times with $i$. After this, the OLBoost does not interfere with tree growth. The instance is then used to update the leaf statistics with a normal weight of 1. Then, a split attempt if performed.

Considering the time costs of using OLBoost, we have some different scenarios. If we compare using it with using a MC or NB predictor, then the OLBoost has time costs associated with making a prediction (which will vary according to the predictor inside the OLBoost), computing $\lambda$, sampling $w$ and updating its statistics. On the other hand, if we compared against using an ANB, then there would be no additional prediction costs since ANB has to perform a prediction to see which predictor (MC or NB) is performing better. Memory costs will increase according to the predictor used inside the OLBoost, being more costly when using a NB or ANB. However, when using ensembles to improve prediction performance, both memory cost increase around $m$ times When the models are sequentially updated, the processing cost also increases by around $m$ times, where $m$ is the number of models inside the ensemble. Nonetheless, when predictive performance needs to be maximised and the other resources are available, OLBoost coupled with an ODT can be used in ensembles.

V Experimental setup

To evaluate the impact of the OLBoost in the VFDT and SVFDTs, eleven benchmark datasets, commonly used in data stream mining experiments, were selected: agrawal, cover_type, electricity, hyper, led24, poker, rbf (with 500k and 10 features, 1M instances and 10 features, and 250k instances and 50 features), sea and usenet. In all the cases, the prequential evaluation scheme was employed for evaluating the algorithms [7].

The VFDT and SVFDTs used ANB and $GP$ and $\tau$ were varied: $GP\in(100,200,400,800,1000)$ and $\tau\in(0.01,0.05,0.10,0.15,0.20)$. Likewise, $min_{\lambda}$ and $max_{\lambda}$ were varied, with performance increasing as $max_{\lambda}$ increased. Based on empirical evaluations the best all around results plateaued for around $min_{\lambda}=1$ and $max_{\lambda}=12$, hereby recommended as default values.

We evaluated the compared algorithms in terms of predictive performance, running time, and the amount of required memory. The predictive performance was measured in terms of accuracy. We accounted for the total running time of the algorithms in seconds and measured the final size of the models (in MB) at the end of the data streams.

All algorithms were implemented in Python 3.7111https://www.python.org/ and Cython222http://cython.org/, a superset of the Python language that allows code to be written in Python and compiled to C extensions, and are available online at 333https://github.com/vturrisi/pystream.

VI Results and Discussions

Fig. 2 presents the boxplots and violin plots of the performance metrics for each dataset separately while varying $GP$ and $\tau$. The violins represent the algorithms without using OLBoost, while the overlapped boxplots represent the same algorithm with OLBoost. As previously discussed, the values of $min_{\lambda}$ and $max_{\lambda}$ were kept constant during the experiments.

Considering accuracy, it is possible to see that excluding agrawal and usenet datasets, using OLBoost increases the median performance. Moreover, OLBoost outperforms the traditional ODT variants in all hyperparameter configurations for the cover_type and electricity datasets. While the traditional ODT algorithm can reach similar predictive performance in some datasets by selecting high values for $\tau$ (creating a deeper tree), using OLBoost grants higher or very similar performance while using a much shallower tree.

Concerning memory usage, OLBoost increases consumption in all cases, but this was expected since it essentially doubles the memory sizes of the leaves. This fact can, however, be neglected for most of the applications, given that our proposal reaches higher predictive performances than standard ODT algorithms. OLBoost can also use shallower structures while maintaining competitive accuracy. In the future, we intend to evaluate new strategies for tree growth stopping based on the measured accuracy. This fact could enhance the advantages of our proposal against other algorithms.

For time costs, a small variation is perceived, but without an impacting increase. Given the previously discussed aspects, settings which lead OLBoost to structures with smaller depths would be preferred to decrease time processing costs. The robustness of our proposal accuracy-wise enables this kind of balance between size and performance. Hence, we advise using OLBoost while tuning ODT’s hyperparameters to create smaller trees maximising the efficiency between predictive performance and memory consumption.

It is also worth mentioning the SVFDT variants when coupled with OLBoost were able to obtain predictive performance comparable to those of VFDT. Nonetheless, we limited our analysis for pairwise comparison of tree settings, i.e., we compared the performance of the same hyperparameter set using or not OLBoost. An interesting venue for the future evaluation would be comparing the performance obtained by SVFDT with OLBoost against the traditional VFDT algorithm. This could lead to more accurate decision models that are restrictive in the usage of computational resources.

We also evaluated the benefits of using OLBoost as the tree structures grow. Fig. 3 presents the average relative accuracy considering all trees when using OLBoost sorted by tree size. The relative accuracy is calculated by dividing the accuracy of the OLBoost-based models by the accuracy of their standard counterparts. Therefore, relative accuracies greater than $1$ implies that our proposal led to higher accuracies, whereas the contrary holds true to values below $1$. As the trees increase, the gains obtained by OLBoost become smaller. We expected such behaviour since the prediction models become more specialised in the incoming concepts as they process more examples. However, considering all ODTs evaluated and the datasets, OLBoost always presents a gain in accuracy.

Lastly, the statistical difference between the six algorithm combinations (VFDT, SVFDT-I, SVFDT-II, and their OLBoost variants: O_VFDT, O_SVFDT-I, and O_SVFDT-I) was assessed using the Wilcoxon signed-rank test [15]. We used the results obtained by each pair of algorithms, considering all the evaluated hyperparameters. Thus, we used a sample size of $25$ (the number of different evaluated hyperparameter combinations per dataset) for each performed test. First, a two-sided test was computed to verify whether the median of the differences between each algorithm pair was zero (null hypothesis) or not (alternative hypothesis). In case the alternative hypothesis was observed, an additional one-sided test was performed to verify whether the median of the differences was positive or negative. Therefore, we can state for each algorithm pair $a$ and $b$, the number of datasets $a$ was statistically better than $b$, and vice versa. We present the observed results in Tables I(a), I(b), and I(c).

Considering accuracy (Table I(a)), the O_VFDT was statistically better than the other algorithms more times than any other algorithm. Moreover, we can also observe that by adding OLBoost to the SVFDT-I and SVFDT-II, both algorithms were able to surpass the accuracy achieved by even the best performing technique (VFDT). From a memory perspective (Table I(b)) it is possible to see that adding OLBoost to any of the algorithms statistically increases memory costs. Nonetheless, O_SVFDT-I and O_SVFDT-II were still able to statistically consume less memory than the VFDT in six and five datasets, respectively. Considering time (Table I(c)), OLBoost always increases computational costs regardless of the base ODT. Despite being slower, the real difference values are around 10 seconds throughout processing the whole stream. This amount can be neglected depending on the type of data stream application.

VII Conclusion and Future Work

This work presented a novel algorithm to increase the predictive performance of ODTs for data stream mining. We considered a large and varied set of benchmark datasets to compare our proposal against traditional ODT algorithms. According to the experimental results, OLBoost is able to increase accuracy performance. Additionally, OLBoost produced trees shallower and with better predictive performance better than the other algorithms, reducing the memory needed, speeding up the test and potentially improving the model’s interpretability. Reducing model size also helps to combat overfitting. In addition, we observed that variants of SVFDT with OLBoost were capable of presenting predictive performance values competitive to those obtained by the traditional VFDT. As future work, we intend to evaluate the possibility of using OLBoost in a restricted set of leaves to increase predictive performance while reducing memory costs. Moreover, the evaluation of OLBoost in ensemble algorithms can also be explored. The adaptation of OLBoost to regression tasks is another possible venue for future research. Finally, our proposal can be extended to other online prediction algorithms, such as k-Nearest Neighbours.