Distribution-Free One-Pass Learning

TL;DR

This paper introduces DFOP, a distribution-free online learning method capable of updating models with single-pass data scans, handling distribution changes without prior knowledge, and providing theoretical guarantees of convergence.

Contribution

The paper presents DFOP, a novel distribution-free one-pass learning algorithm that adapts to changing data distributions in an online setting without needing prior distribution information.

Findings

DFOP effectively handles distribution changes during data accumulation.

Theoretical guarantees show error decreases until convergence.

Experimental results validate DFOP's performance in regression and classification.

Abstract

In many large-scale machine learning applications, data are accumulated with time, and thus, an appropriate model should be able to update in an online paradigm. Moreover, as the whole data volume is unknown when constructing the model, it is desired to scan each data item only once with a storage independent with the data volume. It is also noteworthy that the distribution underlying may change during the data accumulation procedure. To handle such tasks, in this paper we propose DFOP, a distribution-free one-pass learning approach. This approach works well when distribution change occurs during data accumulation, without requiring prior knowledge about the change. Every data item can be discarded once it has been scanned. Besides, theoretical guarantee shows that the estimate error, under a mild assumption, decreases until convergence with high probability. The performance of DFOP for both regression and classification are validated in experiments.

Tables

Table 1. Table 1: Performance comparison in terms of mean and standard deviation of accuracy (both in percents). Bold values indicates the best performance. Besides, ∙ ∙ \bullet ( ∘ \circ ) indicates that DFOP is significantly better (worse) than the compared method (paired t 𝑡 t -tests at 95% significance level). And Win/ Tie/ Loss are summarized in the last row.

Dataset	SVM-win	1NN-win	SVM-fix	SVM-ada	DWM	RLS	DFOP
SEA	73.94 $\pm$ 0.12$\bullet$	77.27 $\pm$ 0.04$\bullet$	86.19 $\pm$ 0.06$\bullet$	83.47 $\pm$ 0.09$\bullet$	87.04 $\pm$ 0.03$\bullet$	84.54 $\pm$ 0.47$\bullet$	87.99 $\pm$0.05
hyperplane	83.74 $\pm$ 0.03$\bullet$	70.66 $\pm$ 0.03$\bullet$	87.98 $\pm$ 0.03$\bullet$	81.94 $\pm$ 0.07$\bullet$	88.36 $\pm$ 0.25$\bullet$	69.67 $\pm$ 1.40$\bullet$	90.14 $\pm$0.05
1CDT	98.71 $\pm$ 0.05$\bullet$	99.96 $\pm$ 0.07	99.77 $\pm$ 0.06$\bullet$	99.77 $\pm$ 0.08$\bullet$	99.90 $\pm$ 0.09$\bullet$	98.79 $\pm$ 1.65$\bullet$	99.97 $\pm$0.05
2CDT	94.86 $\pm$ 0.06$\bullet$	94.62 $\pm$ 0.10$\bullet$	95.19 $\pm$ 0.13$\bullet$	95.18 $\pm$ 0.15$\bullet$	90.21 $\pm$ 0.67$\bullet$	62.24 $\pm$ 0.23$\bullet$	96.36 $\pm$0.09
1CHT	98.75 $\pm$ 0.18$\bullet$	99.81 $\pm$ 0.22	99.63 $\pm$ 0.17$\bullet$	99.63 $\pm$ 0.18$\bullet$	99.69 $\pm$ 0.26$\bullet$	98.49 $\pm$ 1.61$\bullet$	99.84 $\pm$0.16
2CHT	87.70 $\pm$ 0.04$\bullet$	85.69 $\pm$ 0.05$\bullet$	89.48 $\pm$ 0.12$\bullet$	88.89 $\pm$ 0.13$\bullet$	85.92 $\pm$ 0.72$\bullet$	62.57 $\pm$ 0.23$\bullet$	89.91 $\pm$0.07
1CSurr	97.99 $\pm$ 0.04$\bullet$	98.12 $\pm$ 0.11$\bullet$	94.24 $\pm$ 1.08	93.56 $\pm$ 1.08	96.31 $\pm$ 0.50$\circ$	67.82 $\pm$ 0.22$\bullet$	93.24 $\pm$1.44
UG-2C-2D	94.47 $\pm$ 0.13$\bullet$	93.55 $\pm$ 0.16$\bullet$	95.41 $\pm$ 0.10$\bullet$	94.92 $\pm$ 0.12$\bullet$	95.59 $\pm$ 0.11	67.02 $\pm$ 1.46$\bullet$	95.59 $\pm$0.10
UG-2C-3D	93.60 $\pm$ 0.73$\bullet$	92.83 $\pm$ 0.93$\bullet$	95.05 $\pm$ 0.64$\bullet$	94.48 $\pm$ 0.71	95.14 $\pm$ 0.62	61.95 $\pm$ 2.60$\bullet$	95.37 $\pm$0.61
UG-2C-5D	74.82 $\pm$ 0.45$\bullet$	88.04 $\pm$ 0.42$\bullet$	91.74 $\pm$ 0.26$\bullet$	90.37 $\pm$ 0.35$\bullet$	92.82 $\pm$ 0.23$\circ$	81.20 $\pm$ 2.42$\bullet$	92.51 $\pm$0.25
MG-2C-2D	90.20 $\pm$ 0.07$\circ$	87.84 $\pm$ 0.09$\bullet$	84.98 $\pm$ 0.06$\bullet$	84.22 $\pm$ 0.06$\bullet$	90.15 $\pm$ 0.06$\circ$	57.18 $\pm$ 3.66$\bullet$	85.06 $\pm$0.06
G-2C-2D	95.54 $\pm$ 0.01$\bullet$	99.61 $\pm$ 0.00$\circ$	95.41 $\pm$ 0.01$\bullet$	95.26 $\pm$ 0.02$\bullet$	95.82 $\pm$ 0.02	95.84 $\pm$ 0.01	95.83 $\pm$0.02
Chess	69.67 $\pm$ 1.51$\bullet$	79.58 $\pm$ 0.54	77.73 $\pm$ 1.56$\bullet$	69.18 $\pm$ 3.65$\bullet$	73.77 $\pm$ 0.66$\bullet$	78.70 $\pm$ 0.83$\bullet$	79.15 $\pm$0.62
Usenet-1	68.92 $\pm$ 1.12	65.36 $\pm$ 1.55$\bullet$	64.18 $\pm$ 2.24$\bullet$	67.68 $\pm$ 1.86$\bullet$	64.43 $\pm$ 4.53$\bullet$	60.65 $\pm$ 0.53$\bullet$	69.20 $\pm$0.68
Usenet-2	74.44 $\pm$ 0.71$\bullet$	71.03 $\pm$ 0.60$\bullet$	73.99 $\pm$ 0.69$\bullet$	72.64 $\pm$ 0.84$\bullet$	73.37 $\pm$ 0.93$\bullet$	73.16 $\pm$ 0.67$\bullet$	75.60 $\pm$0.57
Luxembourg	88.57 $\pm$ 0.28$\bullet$	77.51 $\pm$ 0.44$\bullet$	98.25 $\pm$ 0.19$\bullet$	97.43 $\pm$ 0.42$\bullet$	92.61 $\pm$ 0.40$\bullet$	99.06 $\pm$ 0.14$\bullet$	99.09 $\pm$0.14
Spam	83.91 $\pm$ 2.20$\bullet$	93.43 $\pm$ 0.82$\bullet$	92.44 $\pm$ 0.80$\bullet$	91.01 $\pm$ 0.94$\bullet$	91.49 $\pm$ 1.09$\bullet$	94.46 $\pm$ 0.16$\bullet$	94.77 $\pm$0.26
Weather	68.54 $\pm$ 0.55$\bullet$	72.64 $\pm$ 0.25$\bullet$	67.79 $\pm$ 0.65$\bullet$	77.26 $\pm$ 0.33$\bullet$	70.86 $\pm$ 0.42$\bullet$	78.35 $\pm$ 0.18$\bullet$	79.23 $\pm$0.12
Powersupply	73.33 $\pm$ 0.25$\bullet$	72.42 $\pm$ 0.21$\bullet$	71.17 $\pm$ 0.15$\bullet$	69.39 $\pm$ 0.17$\bullet$	72.18 $\pm$ 0.29$\bullet$	69.67 $\pm$ 0.64$\bullet$	80.46 $\pm$0.04
Electricity	74.20 $\pm$ 0.08$\bullet$	85.33 $\pm$ 0.09$\circ$	62.01 $\pm$ 0.59$\bullet$	58.69 $\pm$ 0.58$\bullet$	78.60 $\pm$ 0.41$\bullet$	74.20 $\pm$ 0.63$\bullet$	76.94 $\pm$0.26
DFOP W/ T/ L	18/ 1/ 1	14/ 4/ 2	19/ 1/ 0	18/ 2/ 0	14/ 3/ 3	19/ 1/ 0	-

Table 2. Table 2: Datasets, the number of data items between consecutive distribution change, theoretical recommended value and empirical appropriate value for forgetting factor are listed below, and the last column provides relative proportion between theoretical recommended value and empirical appropriate value.

Dataset	$T_0$	$1/T_0$	$\mu$	$\mu T_0$	Dataset	$T_0$	$1/T_0$	$\mu$	$\mu T_0$
1CDT	400	2.50E-03	1.00E-02	4	UG-2C-2D	1,000	1.00E-03	1.00E-03	1
2CDT	400	2.50E-03	1.00E-02	4	UG-2C-3D	2,000	5.00E-04	1.00E-03	2
1CHT	400	2.50E-03	1.00E-02	4	UG-2C-5D	2,000	5.00E-04	1.00E-03	2
2CHT	400	2.50E-03	1.00E-02	4	MG-2C-2D	2,000	5.00E-04	1.00E-03	2
1CSurr	600	1.67E-03	5.00E-03	3	G-2C-2D	2,000	5.00E-04	1.00E-03	2
hyperplane	9,000	1.11E-04	2.00E-03	18	SEA	10,000	1.00E-04	1.00E-03	10

Table 3. Table 3: Mean and standard deviation of MSE over 10 trails, the lower the better, and the all the values have been multiplied by 100. Besides, ∙ ∙ \bullet ( ∘ \circ ) indicates that DFOP is significantly better (worse) than the compared method (paired t 𝑡 t -tests at 95% significance level). Note that the comparisons are unfair to DFOP, because DFOP is able to only scan data once, whereas most compared methods not.

Methods	hyperplane	SRU-1	SRU-2	Debutanizer
RLS	2.291(.045)$\bullet$	0.306(.006)$\bullet$	0.392(.005)$\bullet$	2.467(.003)$\bullet$
OB	1.914(.004)$\bullet$	0.051(.000)	0.142(.000)$\bullet$	0.577(.000)$\bullet$
AddExp.C	0.730(.002)$\bullet$	0.049(.001)$\circ$	0.139(.001)$\bullet$	1.283(.000)$\bullet$
EOS-ELM	1.903(.002)$\bullet$	0.050(.000)	0.142(.001)$\bullet$	1.057(.000)$\bullet$
Learn++.NSE	1.662(.009)$\bullet$	0.098(.028)$\bullet$	0.232(.024)$\bullet$	1.040(.000)$\bullet$
OAUE	1.959(.002)$\bullet$	0.047(.000)$\circ$	0.131(.001)$\bullet$	0.731(.008)$\bullet$
DFOP	0.619(.048)	0.063(.004)	0.064(.004)	0.360(.057)

Table 4. Table 4: Basic information of various datasets with distribution change

Dataset	#instance	#dim	Dataset	#instance	#dim
SEA	50,000	3	Chess	533	7
hyperplane	90,000	10	Usenet-1	1,500	100
1CDT	16,000	2	Usenet-2	1,500	100
2CDT	16,000	2	Luxembourg	1,900	32
1CHT	16,000	2	Spam	9,324	500
2CHT	16,000	2	Weather	18,159	8
1CSurr	55,283	2	Powersupply	29,928	2
UG-2C-2D	100,000	2	Electricity	45,312	8
UG-2C-3D	200,000	3	MG-2C-2D	200,000	2
UG-2C-5D	200,000	5	G-2C-2D	200,000	2

Equations

$$y(t)=\mathbf{x}(t{)}^{\mathrm{T}}\mathbf{w}(t-1)+ϵ(t)$$

$$\hat{\mathbf{w}}(t)={\mathrm{argmin}}_{\mathbf{w}\in {\mathbb{R}}^d}\sum _{i=1}^t(\prod _{j=i+1}^t\lambda (j)){\left[y(i)-\mathbf{x}(i{)}^{\mathrm{T}}\mathbf{w}\right]}^2,$$

$$\hat{\mathbf{w}}(t)={\mathrm{argmin}}_{\mathbf{w}\in {\mathbb{R}}^d}\sum _{i=1}^t{\lambda }^{t-i}{\left[y(i)-\mathbf{x}(i{)}^{\mathrm{T}}\mathbf{w}\right]}^2,$$

$$[\mathbf{w}(t){]}_{opt}={\left[\sum _{i=1}^t{\lambda }^{t-i}\mathbf{x}(i)\mathbf{x}(i{)}^{\mathrm{T}}\right]}^{-1}\left[\sum _{i=1}^t{\lambda }^{t-i}\mathbf{x}(i)y(i)\right]$$

$$\mathbf{w}(t)=\mathbf{w}(t-1)+\mathbf{s}(t),t\ge 1$$

$$\mathbb{E}[\exp (sX)]\le \exp (\frac{s^2{\sigma }^2}{2}),\forall s\in \mathbb{R}$$

$$\forall t\ge 1:\mathbb{E}\left\{\exp \{\Vert \mathbf{x}(t){\Vert }^2/{\gamma }_t^2\}\right\}\le \exp \{1\}$$

$$\mathrm{Pr}\left\{\Vert S_N\Vert \ge \sqrt{2}(1+\gamma )\sqrt{\sum _{i=1}^N{\sigma }_i^2}\right\}\le \exp \left\{-{\gamma }^2/3\right\},$$

$$\Vert \sum _{k=1}^t(1-\mu {)}^{t-k}Y(k)\mathbf{x}(k)\Vert \le \sqrt{2}(1+\sqrt{3\ln (2t/\delta )})\cdot {\gamma }^{\ast }(\Vert Y(0)\Vert +Z^{\ast }){\mu }^{-\frac{1}{2}}$$

$$\Vert \sum _{k=1}^t(1-\mu {)}^{t-k}\mathbf{x}(k)ϵ(k)\Vert \le 2\sqrt{2}\left(1+\sqrt{3\ln \frac{2t}{\delta }}\right){\sigma }^{\ast }{\mu }^{-\frac{1}{2}}$$

$$\begin{array}{rl}\Vert \mathbf{w}(t)-\hat{\mathbf{w}}(t)\Vert & \le K\{(1-\mu {)}^t\Vert R(0)\Vert \Vert \tilde{\mathbf{w}}(0)\Vert +\sqrt{2}(1+\sqrt{3\ln (2t/\delta )})\\ & \cdot [2{\sigma }^{\ast }{\mu }^{1/2}+{\gamma }^{\ast }(\Vert R(0)\Vert +{x^{\ast }}^2){\mu }^{-1/2}]\}\end{array}$$

$$\Vert \mathbf{w}(t)-\hat{\mathbf{w}}(t)\Vert =\tilde{O}(\sqrt{\mu }+\frac{1}{\sqrt{\mu }})+o(1),w.h.p.$$

$$\text{AA}(t)=\sum _{i=1}^t\mathbb{I}[\hat{y}(i)=y(i)]/t,$$

$$(1-\mu {)}^t=e^{t\ln (1-\mu )}\approx e^{-t\mu }=e^{-t/T_0},$$

$$\hat{\mathbf{w}}(t)={\mathrm{argmin}}_{\mathbf{w}\in {\mathbb{R}}^d}\sum _{i=1}^t(\prod _{j=i+1}^t\lambda (j)){\left[y(i)-\mathbf{x}(i{)}^{\mathrm{T}}\mathbf{w}\right]}^2$$

$$[A+BCD{]}^{-1}=A^{-1}-A^{-1}B[D{A}^{-1}B+C^{-1}{]}^{-1}D{A}^{-1}$$

$$\begin{array}{rl}& \left\{A^{-1}-A^{-1}B[D{A}^{-1}B+C^{-1}{]}^{-1}D{A}^{-1}\right\}[A+BCD]\\ =& I+A^{-1}BCD-A^{-1}B[D{A}^{-1}B+C^{-1}{]}^{-1}D\\ & -A^{-1}B[D{A}^{-1}B+C^{-1}{]}^{-1}D{A}^{-1}BCD\\ =& I+A^{-1}B[D{A}^{-1}B+C^{-1}{]}^{-1}\{0\}=I\end{array}$$

$$\hat{\mathbf{w}}(t)={\left[\sum _{i=1}^t\Lambda (i,t)\mathbf{x}(i)\mathbf{x}(i{)}^{\mathrm{T}}\right]}^{-1}\left[\sum _{i=1}^t\Lambda (i,t)\mathbf{x}(i)y(i)\right]$$

$$\bar{R}(t)=\lambda (t)\bar{R}(t-1)+\mathbf{x}(t)\mathbf{x}(t{)}^{\mathrm{T}}$$

$$\begin{array}{rl}\hat{\mathbf{w}}(t)& ={\bar{R}}^{-1}(t)\left[\lambda (t)\sum _{i=1}^{t-1}\Lambda (i,t-1)\mathbf{x}(i)y(i)+\mathbf{x}(t)y(t)\right]\\ & ={\bar{R}}^{-1}(t)\left[\lambda (t)\bar{R}(t-1)\hat{\mathbf{w}}(t-1)+\mathbf{x}(t)y(t)\right]\\ & ={\bar{R}}^{-1}(t)\left[(\bar{R}(t)-\mathbf{x}(t)\mathbf{x}(t{)}^{\mathrm{T}})\hat{\mathbf{w}}(t-1)+\mathbf{x}(t)y(t)\right]\\ & =\hat{\mathbf{w}}(t-1)+{\bar{R}}^{-1}(t)\mathbf{x}(t)\left[y(t)-\mathbf{x}(t{)}^{\mathrm{T}}\hat{\mathbf{w}}(t-1))\right]\end{array}$$

$$\begin{array}{rl}P(t)& =\frac{1}{\lambda (t)}P(t-1)-\frac{1}{\lambda (t)}P(t-1)\mathbf{x}(t)\\ & \cdot {\left[\frac{1}{\lambda (t)}\mathbf{x}(t{)}^{\mathrm{T}}P(t-1)\mathbf{x}(t)+1\right]}^{-1}\frac{1}{\lambda (t)}\mathbf{x}(t{)}^{\mathrm{T}}P(t-1)\\ & =\frac{1}{\lambda (t)}\left[P(t-1)-\frac{P(t-1)\mathbf{x}(t)\mathbf{x}(t{)}^{\mathrm{T}}P(t-1)}{\lambda (t)+\mathbf{x}(t{)}^{\mathrm{T}}P(t-1)\mathbf{x}(t)}\right]\end{array}$$

$$\mathbb{E}[\exp (sX)]\le \exp (\frac{s^2{\sigma }^2}{2}),\forall s\in \mathbb{R}$$

$$\mathbb{E}[\exp (sX)]\le \exp (\frac{s^2{\lambda }^2}{2}),\forall |s|\le \frac{1}{\lambda }.$$

$$\mathbb{E}\exp (\Vert \mathbf{x}\Vert )\le 2\exp \left\{\frac{d^2{\sigma }^2}{2}\right\}$$

$$\mathbb{E}[e^{sZ}]$$

$$\le$$

$$=$$

$$\forall t\ge 1:\mathbb{E}\left\{\exp \{\Vert \mathbf{x}(t){\Vert }^2/{\gamma }_t^2\}\right\}\le \exp \{1\}$$

$$\mathbb{E}\exp (\Vert \mathbf{x}(t)\Vert )\le 2\exp \left\{\frac{d^2{\sigma }_t^2}{2}\right\}$$

$$\mathbb{E}[\exp \{s(\Vert \mathbf{x}(t){\Vert }^2-\mathbb{E}[\Vert \mathbf{x}(t){\Vert }^2])\}]\le \exp \{s^{2}16{\sigma }_t^2/2\},\forall |s|\le \frac{1}{4{\sigma }_t}$$

$$\mathbb{E}[\exp \{s\Vert \mathbf{x}(t){\Vert }^2\}]\le \exp \{8s^2{\sigma }_t^2+s\mathbb{E}[\Vert \mathbf{x}(t){\Vert }^2]\},$$

Full text

\cortext

[cor1]Corresponding author. Email: zhouzh@nju.edu.cn

1 Introduction

With a rapid growth in collecting data, the volume of data generated makes it a challenge for traditional machine learning approaches. The main challenges are multi-faced, in general, accumulating and evolving properties are two of the most troublesome issues.

For the accumulating property, apparently, it’s impractical to store all the data entirely due to the limitation of memory and computation resources. Hence, an offline approach is not suitable any more in such tasks. Only in an online processing paradigm can prediction models be trained and updated incrementally. It’s also worthy to mention that an online streaming approach is called one-pass when it requires going through the training data only once without storing the entire dataset. The reason to pursue one-pass property is due to the fact that sometimes the raw data is discarded or no longer accessible after being processed. A one-pass approach guarantees the learning process be independent with the volume of data stream. Apparently, it is much more demanding and difficult.

Furthermore, the evolving nature of data stream also makes it challenging to directly apply traditional machine learning approaches because it’s not reasonable to assume that the current and future data are coming from the same distribution any more. In a non-stationary environment, which is very common in data generation process, the distribution underlying is likely to dynamically change over time. For instance, the clicking information collected in recommendation system is certainly evolving because customers’ interests probably change when looking through the product pages. Another example is credit scoring, the criteria of credit granting should properly alter since a changing economic conditions would have a great influence on people’s manner. Such phenomena above are typical examples of distribution change. Under this scenario, the performance of traditional approaches dramatically drop down and thus are not empirically and theoretically suitable for these tasks.

To simultaneously address these two issues, in this work, we propose DFOP, a distribution free one-pass learning approach to deal with data stream emerging distribution change under one-pass constraints. The advantages of our approach are following: firstly, by recursively solving the target, we guarantee that only one time would data stream be gone through. Secondly, based on a forgetting mechanism, the loss of older data is discounted without explicitly modelling the dynamics or assuming prior information about distribution change. In streaming regression, a theoretical guarantee is presented showing that the estimate error of dynamic concept would decrease until convergence with high probability. Meanwhile, empirical experiments on both synthesis and real-world datasets indicate the effectiveness and practicability of DFOP in both regression and classification scenarios.

The rest of this paper is organized as follows. In Section 2, we briefly review of some related work. Then, static scenario model is introduced in preliminaries. Next, in the non-stationary environment, DFOP is presented to handle dynamics in regression and classification scenarios. Both theoretical guarantee and empirical effectiveness have been examined. Finally, we conclude the paper.

2 Related Work

Online and One-Pass Algorithms. With a rapid growth of data volumn and velocity, it’s no longer practicable to adopt offline mode algorithm for streaming data learning tasks. Hence, online style algorithms become gradually attractive which update the current model with the most recent examples. In general, they are error driven, updating the current model depending on whether the current example is misclassified GZB*+* (14). Representative algorithms include Perceptron Ros (58), Winnow Lit (87) and many other variations. Besides, a paradigm of “prediction with expert advice” CBL (06) also inspires some interesting works, such as AddExpKM (05) and DWMKM (03, 07). Most of those approaches require to store the entire or partial training data and scan data items multiple times. Recently, one-pass algorithms gradually draw more attentions demanding that each data item should be processed only once. Concretely speaking, after a data item has been processed and relevant statistics have been stored, the raw data item should be discarded and never be accessed any more. Obviously, one-pass constraints impose a higher degree of difficulty on algorithm design. Some efforts have been devoted WBSD (16).

Nonstationary Learning. Owing to the effectiveness and simplicity, sliding window is usually adopted to handle data stream with distribution change. It only uses a fixed or variable number of recent data which are the most informative for current prediction Lit (87); Kli (04). Usually, the model built is updated following two processes: one is a learning process, i.e., updating the model based on the new coming data; the other one is a forgetting process, i.e., discarding data items that are moving out of the window GZB*+* (14). However, how to choose an appropriate window size is of great importance which now mainly depends on heuristics to a certian extent. Some efforts have been paid to select window size adaptively Kli (04); GMCR (04); KZ (09). The common strategy to adjust window size is based on the performance or estimate of generalization error. SVM-ada Kli (04) presents a theoretically supported approach , however, the computational efficiency issue makes it not practical in real-world applications.

Our proposed approach DFOP, short for Distribution-Free One-Pass, is a one-pass style algorithm, i.e., it could guarantee that only one time will data items been gone through. Besides, DFOP is distribution-free, i.e., different from those traditional approaches dealing with distribution change, we did not explicitly model the dynamics, and no prior information about distribution change is assumed.

3 Preliminaries

In this part, streaming regression model in a static scenario is briefly introduced.

In a streaming scenario, we denote a labeled dataset as $\{\mathbf{x}(t),y(t)\}$, where $\mathbf{x}(t)$ is the feature of the $t$-th instance and $y(t)$ is a real-valued output. Furthermore, we assume a linear model as follows,

[TABLE]

where $\{\epsilon(t)\}$ is the noise sequence, $\{\mathbf{w}(t)\}$ is what we desire to estimate.

When in a static scenario, the sequence $\{\mathbf{w}(t)\}$ is a constant vector denoted as $\mathbf{w}_{0}$. Then, the least square could be adopted to minimize the residual sum of squares, which has a close-form solution. However, it fails when adding an online/one-pass constraint which demands the raw item is discarded after it has been processed. Recursive least square (RLS) and stochastic gradient descent (SGD) are two typical approaches to solve this problem in an online paradigm.

When in a non-stationary environment, especially when the distribution underlying changes, traditional approaches are not suitable since we could never expect the typical i.i.d assumption continue to work any longer. In the next sections, we propose to handle this scenario based on exponential forgetting mechanism without explicitly modelling the evolution of data stream, and theoretical support and empirical demonstration are presented.

In the following, $\lVert\cdot\rVert$ denotes the $\ell_{2}$-norm in $\mathbb{R}^{n}$ space. Meanwhile, for a bounded real-valued sequence $\{x(t)\}$, $x^{*}$ denotes the upper bound of sequence, namely, $x^{*}=\sup_{t=1,2,\cdots}x(t)$.

4 Distribution Free One-Pass Learning

Since the sequence $\{\mathbf{w}(t)\}$ is changing over time in a dynamic environment, it is no longer reasonable to estimate current (i.e., at time $t$) concept via methods introduced previously. Instead, we introduce a sequence of discounted factors $\{\lambda(t)\}$ to downweight the loss of older data as follows,

[TABLE]

where $\lambda(i)\in(0,1)$ is a discounted factor to smoothly put less weight on older data. The intuition can be more easily obtained if we simplify all $\lambda(i)$ as a constant $\lambda\in(0,1)$, then the target function is,

[TABLE]

And the quantity $\mu\triangleq 1-\lambda$ is named as forgetting factor Hay (08). The value of forgetting factor is, as a matter of fact, a trade-off between stability of past condition and sensitivity of future evolution.

It should be pointed out that the forgetting mechanism based on exponential forgetting factor could be also considered as a continuous analogy to sliding window approach to some extent. The older data items with a small enough weight can be somehow thought as exclusion from the window. Some. More discussions on relation with window size and forgetting factor are provided in Section 5.4.

4.1 Algorithm

For the optimization problem proposed in (3), obviously, by taking derivative of the target function, we can directly obtain the optimal solution in a closed-form,

[TABLE]

However, above expression is an off-line estimate, namely, all the data items ahead of $t$ are needed. Instead of repeatedly solving (4), we estimate the underlying concept by adding a correction term to the previous estimate based on the information of new coming data item. With the forgetting factor recursive least square method Hay (08), we could solve the target (3) in a one-pass paradigm. And to the best of our knowledge, this is the very first time to adopt traditional forgetting factor RLS to deal with such tasks with distribution change under the one-pass constraints. And we named this as DFOP(short for Distribution-Free One-Pass) summarized in Algorithm 1.

Besides, it should be pointed out that $\{\lambda(t)\}$ is by no means necessary chosen as a constant, we provide a generalized DFOP (short as G-DFOP) for a dynamic discount factor sequence $\{\lambda(t)\}$, corresponding to target in (11), which is also provably a one-pass algorithm. Detailed proofs are provided in Section 1 of supplementary material.

For the classification scenario, $y(t)$ is no longer a real-valued output but a discrete value, and we assume $y(t)\in\{-1,+1\}$ for convenience. A slight modification on original output step is applied in classification, where the effectiveness is empirically validated in the next section.

Assuming that the feature is $d$-dimension, we only need to keep $P(t)\in\mathbb{R}^{d\times d}$ in memory during the algorithm processing procedure. In other words, the storage is always $O(d^{2})$, which is independent to the number of training examples. Besides, at the $t$-th time stamp, the update of $\hat{\mathbf{w}}(t)$ is unrelated to the previous data items, namely every data item can be discarded once it has been scanned.

4.2 Theoretical Guarantee

In this section, we develop an estimate error bound in a non-stationary regression scenario.

Consider the additive model of drift in sequence $\{\mathbf{w}(t)\}$,

[TABLE]

We assume that the adding term $\{\mathbf{s}(t)\}$ is an $E$-valued martingale-difference $d$-dimension sub-Gaussian vector sequence, with corresponding variance proxy sequence $\{\sigma_{t}\}$, whose formal definition will be given following. The $E$-valued martingale-difference assumption is reasonable, in fact, in many real-world application, the drift of concepts are usually independent.

Similar to the analysis in GLP (93), we relax the assumptions to be more realistic in real-world applications and provide a non-deterministic estimate error bound $\lVert\mathbf{w}(t)-\hat{\mathbf{w}}(t)\rVert$ based on vector concentration, showing that the estimate error is tending to convergence with high probability.

Now we give the formal definitions of sub-Gaussian random variable and random vector.

Definition 1

(sub-Gaussian random variable) A random variable $X\in\mathbb{R}$ is said to be sub-Gaussian with variance proxy $\sigma^{2}$ if $\mathbb{E}[X]=0$ and its moment generating function satisfies

[TABLE]

Definition 2

(sub-Gaussian random vector) A random vector $\mathbf{x}\in\mathbb{R}^{d}=(x_{1},\cdots,x_{d})$ is called sub-Gaussian with variance proxy $\sigma^{2}$ if all its coordinates are sub-Gaussian random variables with variance proxy $\sigma^{2}$.

To exploit concentration property of sub-Gaussian random vector, condition ($\mathcal{C}_{\alpha}[\sigma^{\infty}]$) proposed in Theorem 2.1 of JN (08) shall be satisfied. Thus, first, we show that there exists a bounding sequence $\{\gamma_{t}\}$ for a sub-Gaussian random vector sequence $\{\mathbf{x}(t)\}$.

Lemma 1

For a sub-Gaussian random vector sequence $\{\mathbf{x}(t)\}$ with a variance proxy sequence $\{\sigma_{t}\}$, there exists a corresponding positive bounding sequence $\{\gamma_{t}\}$, such that

[TABLE]

Lemma 1 guarantees the ”light tail” assumption of sub-Gaussian random vector. Then we could apply the following vector concentration, which is a corollary of Theorem 2.1 proposed in JN (08).

Theorem 1

(Corollary of Theorem 2.1 in JN (08)) In an Euclidean space $(\mathbb{R}^{n},\lVert\cdot\rVert_{2})$, let E-valued martingale-difference sub-Gaussian sequence $\xi^{\infty}$ with a corresponding bounding sequence $\sigma^{N}=[\sigma_{1};\cdots;\sigma_{N}]$. Let $S_{N}=\sum_{i=1}^{N}\xi_{i}$, then for all $N\geq 1$ and $\gamma\geq 0$:

[TABLE]

Based on Theorem 23, we could provide Lemma 2 and Lemma 3 to bound a sum of sub-Gaussian random vectors and random variables with exponential decrease, respectively.

Lemma 2

Let $\{\mathbf{x}(t)\}$ be an $E$-valued martingale-difference $d$-dimension sub-Gaussian random vector sequence, with corresponding bounding sequence $\{\gamma_{t}\}$, and $Z(t)\in\mathbb{R}^{d\times d},Y(t)=(1-\mu)Y(t-1)+\mu Z(t),t\geq 1$. Then for $\mu\in(0,1)$, with a probability at least $1-\frac{\delta}{2}$, we have

[TABLE]

where $Z^{*}=\sup_{k=1,\cdots,t}\lVert Z(k)\rVert$ and $\gamma^{*}=\sup_{k=1,\cdots,t}\gamma_{k}$.

Lemma 3

Let $\{\epsilon(t)\}$ be an independent (or $E$-valued martingale-difference) sub-Gaussian random variable sequence, with corresponding bounding sequence (i.e., variance proxy sequence) $\{\sigma_{t}\}$, and $\mathbf{x}(t)\in\mathbb{R}^{d},t\geq 1$. Then for $\mu\in(0,1)$, with a probability at least $1-\frac{\delta}{2}$, we have

[TABLE]

where $\sigma^{*}=\sup_{k=1,\cdots,t}\lVert\mathbf{x}(k)\rVert\cdot\sup_{k=1,\cdots,t}\sigma_{k}$.

Theorem 2

Assume following conditions be satisfied:

• (i)

drift term $\{\mathbf{s}(t)\}$ is an $E$-valued martingale-difference sub-Gaussian random vector sequence, with corresponding bounding sequence $\{\gamma_{t}\}$;

• (ii)

output noise $\{\epsilon(t)\}$ is an independent (or $E$-valued martingale-difference) sub-Gaussian random variable sequence, with corresponding bounding sequence (i.e., variance proxy sequence) $\{\sigma_{t}\}$.

Then with a probability at least $1-\delta$, we have

[TABLE]

where $K=\sup_{k=1,\cdots,t}\lVert P(k)\rVert$, $x^{*}=\sup_{k=1,\cdots,t}\lVert\mathbf{x}(k)\rVert$, $\sigma^{*}=\sup_{k=1,\cdots,t}\lVert\mathbf{x}(k)\rVert\cdot\sup_{k=1,\cdots,t}\sigma_{k}$ and $\gamma^{*}=\sup_{k=1,\cdots,t}\gamma_{k}$.

Remark. The estimate error bound can be decomposed into three parts, i.e., the first one is $(1-\mu)^{t}\lVert R(0)\rVert\lVert\bm{\tilde{\mathbf{w}}}(0)\rVert$, second one is $2\sqrt{2}(1+\sqrt{3\ln 2t/\delta})\sigma^{*}\mu^{1/2}$ and third one is $\sqrt{2}(1+\sqrt{3\ln 2t/\delta})(\|R(0)\|+{x^{*}}^{2})\gamma^{*}\mu^{-1/2}$. Apparently, the first term is decreasing to zero as $t$ increases to infinity, second term is caused by the output noise which shall not be erased, and the third term is introduced by drift of $\mathbf{w}(t)$. Ignoring the poly-logarithmic factors in $t$ and $d$, then, an asymptotic analysis gives the estimate error bound as,

[TABLE]

where we use the $\tilde{O}$ notation to hide constant factors as well as poly-logarithmic factors in $t$ and $d$, and $o(1)$ will exponentially decrease to zero as $t\rightarrow\infty$.

Due to the page limits, we present the proofs of Theorem 1 and Theorem 2 (along with Lemma 1, 2 and 3) in Section 2 and 3 of supplementary material, respectively.

5 Experiments

In this section, we examine the empirical performance of the proposed DFOP on both regression and classification scenarios. Then, we analyze the parameter sensitivity in Section 5.4. However, due to the page limits, only results on the classification scenario are provided, and the regression ones are appended in the supplementary materials.

Moreover, considering that when dealing with real-world datasets, we could not grasp the evolving distribution, specifically, the start and end time of drift, the underlying distribution. As a consequence, it would be very incomplete to analyze the behaviour of algorithms. Hence, both synthesis and real-world datasets are included in the comparison experiments.

5.1 Comparisons Methods

We compare the proposed approach with six common methods on both synthesis and real-world datasets. The comparison methods are (a) RLS, least square approach solved in a recursive manner, (b) Sliding window approach, the classifier is constantly updated by the nearest data samples in the window. Base classifiers are 1NN and SVM, denoted as 1NN-win and SVM-win dSSGB (15), (c) SVM-fix, batch implementation of SVM with a fixed window size SLS (99), (d) SVM-ada,  batch implementation of SVM with an adaptive window size Kli (04), (e) DWM, dynamic weighted majority algorithm, an adaptive ensemble based on the traditional weighted majority algorithm Winnow KM (03, 07).

It’s noteworthy to emphasize that the above comparisons are not all fair enough, because DFOP requires each data item be processed only once. Moreover, DFOP only needs one instance to update the model. Not all comparison methods can meets these two constraints, specifically, 1NN-win, SVM-win, SVM-fix and SVM-ada are window-based algorithms, hence, they are not one-pass. Besides, SVM-fix and SVM-ada are not incremental but updated in a series of batches. DWM is incremental style but not one-pass because it needs to use data to update experts pool in addition.

5.2 Synthetic Datasets

First, we present the performance comparisons over synthetic datasets.

SEA SK (01) consists of three attributes $x_{1},x_{2},x_{3}$, and $0.0\leq x_{i}\leq 10.0$. The target concept is $x_{1}+x_{2}\leq b$, and there are 50,000 instances with 4 stages where $b\in\{7,8,9,9.5\}$.

• -

hyperplane Fan (04), is generated uniformly in a 10-dimensional hyperplane with 90,000 instances in total over 9 different stages.

Besides, another 11 synthesis datasets for binary classification are also adopted. Detailed information are included in the supplementary materials.

The performance is measured by holdout accuracy since underlying joint distribution of synthetic datasets are known. Holdout accuracy is calculated over testing data generated according to the identical distribution as training data at each time stamp. Performance comparisons of seven approaches on SEA and hyperplane datasets are depicted in Figure 1. Since the accuracy curves of SVM-ada, SVM-fix, 1NN-win and SVM-win are so unstable that they would shield all the other curves, we also present a relatively neat figure containing RLS, DWM and DFOP only.

As shown in Figure 1, the accuracy of all algorithms falls rapidly when the underlying distribution emerges abrupt drift, and then will rise up with more data coming. DFOP is significantly better than RLS which is a special case of DFOP, this phenomenon validates the effectiveness of forgetting mechanism. Furthermore, the best two algorithms, obviously, are DFOP and DWM, both of them can converge to new stage quickly. DFOP shows a slightly better performance than DWM, both in slope and asymptote. Moreover, DWM requires to dynamically maintain a set of experts and needs previous data to update experts pool and to decide whether to remove poorly performing experts. On the contrary, DFOP demonstrates a desirable performance requiring to scan each data item only once.

5.3 Real-world Datasets

To valid the effectiveness of DFOP in real-world applications, performance comparisons are presented over 8 real-world datasets. Detailed descriptions are provided in Section 5 of supplementary materials.

In real-world datasets, we can never expect to foreknow the underlying distribution at each data stamp. Thus, it’s not possible to still adopt holdout accuracy as performance measurement. In Table 4, we conduct all the experiments for 10 trails and report the overall mean and standard deviation of predictive accuracy over above real-world datasets as well as other 12 synthesis datasets.

In a total of 20 datasets, the number of instance vary from 533 to at most 200,000. DFOP achieves the best among all approaches in 15 over 20 datasets. Also, in other 5 datasets, DFOP ranks the second or the third. This validates the effectiveness of DFOP, especially under an unfair comparison condition.

Additionally, the robustnessVDG*+* (02) of all these different algorithms are compared. Briefly speaking, for a particular algorithm algo, the robustness $r_{algo}$ is defined as the proportion between its accuracy and the smallest accuracy among all compared algorithms, i.e., $r_{algo}=acc_{algo}/\min_{\alpha}acc_{\alpha}$. Hence, the sum of $r_{algo}$ over all datasets indicates the robustness of for algorithm $algo$. The greater the value of the sum, the better the performance. DFOP achieves the best over 20 datasets, and RLS ranks last as expected since it didn’t consider the evolving distribution in datasets at all. Due to the page limits, detailed robustness comparison results could be found in Section 4.3 in supplementary materials.

5.4 Parameter Study

As stated previously, how to choose an appropriate forgetting factor is an important issue since it reflects a trade-off between stability of past condition and sensitivity to future evolution. To figure out how forgetting factor affects the performance, in classification problem, accumulated accuracy (short as ’AA’) is adopted as a performance measurement in the time series and is defined as,

[TABLE]

where $\mathbb{I}(\cdot)$ is indicator function which takes 1 if $\cdot$ is true, and 0 otherwise, $\hat{y}(i)$ and $y(i)$ are predictive and ground-truth label, respectively. Figure 2 shows the impact of different forgetting factor $\mu$ over four datasets. We notice that the accumulated accuracy of RLS almost decreases all the time. For a relatively small but not zero $\mu$, the performance is satisfying without a significant gap. However, when $\mu$ is too large, say 0.5, the performance is even much worse than RLS. This is consistent with intuition since forgetting factor is so large and older data samples are exponentially downweighted that there are not sufficient effective training samples available to update the model.

Now, here comes the question: how to choose an appropriate forgetting factor to adapt the distribution change in the data stream? To answer this question, let’s recall the target function in (3), when $\mu$ is close to [math], which is often the case in practice and validated in Figure 2, then we have

[TABLE]

where we define $T_{0}=1/\mu$ as forgetting period. The contribution for prediction error of data items older than $T_{0}$ time will be discounted with a weight less than $e^{-1}\approx 36.8\%$ comparing to the current data. As a matter of fact, the forgetting period in forgetting mechanism is pretty similar to the window size in sliding window technique. It can be regarded as a soft relaxation of window size. Consequently, the forgetting factor $\mu$ shall be chosen according to the forgetting period $T_{0}$, where the data distribution should be relatively smooth and stable during this forgetting period.

We validate this idea over synthesis datasets reported in Table 2. Theoretical recommended value $1/T_{0}$ and empirical appropriate value $\mu$ for forgetting factor are provides. Also, the relative proportions between them are calculated. We can see that these two value are very close over all datasets with no more than 20 times difference, even no more than 5 times in most datasets. This supports our strategy in choosing forgetting factor.

Certainly, the drifting properties of real-world datasets are not as clear as synthetic datasets. Nevertheless, we could still infer the forgetting period $T_{0}$ based on the domain knowledge and choose an appropriate value as forgetting factor. For instance, considering the weather forecast dataset, although we cannot foreknow the drifting property of distribution, a relative stable period can still be estimated.

6 Conclusion

In this paper, we proposed an approach based on forgetting mechanism called DFOP handling streaming learning problems with distribution change. The main idea is to downweight the older data items by introducing exponential forgetting factor without considering any prior about drifting information. Meanwhile, DFOP meets the one-pass constraints guaranteeing that only once will the data items be scanned without storing the entire dataset. Hence, DFOP. The storage requirement of DFOP is $O(d^{2})$, where $d$ is the dimension of data, independent from the number of training examples. Both theoretical supports and empirical demonstrations for DFOP are presented to validate its effectiveness and practicality.

Besides, how to efficiently reduce the storage and make DFOP paralleled to adapt a even larger scale real-world applications would be an interesting future work.

Appendix

1 Generalized DFOP and Proofs

1.1 Recursive Algorithm for Dynamic Discounted Factors

When dynamic discounted factors are introduced to downweight the contribution of older data items, the target function can be written as,

[TABLE]

In this part, we provide a provably recursive algorithm to directly solve (11) as shown in Algorithm 2 named generalized DFOP algorithm, short as G-DFOP.

1.2 Proof of Generalized DFOP

In this part, we will prove the consistency between G-DFOP and target function in (11).

Lemma 4

Let $A,B,C$ and $D$ be matrices of compatible dimensions such that the product $BCD$ and the sum $A+BCD$ exist. Then we have

[TABLE]

Proof.

Multiply the right-hand side of (12) by $A+BCD$ from the right, this gives

[TABLE]

∎

For convenience, let $\Lambda(i,t)=\prod_{j=i+1}^{t}\lambda(j)$, then the close-form solution of optimization (11) can be calculated as follows

[TABLE]

Now we will prove that the solution obtained by G-DFOP is equivalent to close-form solution in (13).

Theorem. By the policy in Algorithm 2, we can achieve the same solution as result in (13).

Proof.

Denote $\bar{R}(t)=\sum_{k=1}^{t}\Lambda(i,t)\mathbf{x}(t)\mathbf{x}(t)^{\mathrm{T}}$, obviously

[TABLE]

Then the solution in (13) can be rewritten into the following form:

[TABLE]

Now, we introduce $P(t)=\bar{R}^{-1}(t)$ and then apply Lemma 4 to (1.2), this gives

[TABLE]

Let $L(t)=P(t)\mathbf{x}(t)$, we can obtain the policy described in Algorithm 2.

Remark. Obviously, DFOP in paper is only a special case when fixing discounted factor sequence $\{\lambda(t)\}$ as $(1-\mu)$. Note that, for a simplicity notations in estimate error analysis of DFOP, we slightly modified $L(t)$ and $P(t)$ multiplying by $\mu$.

2 Proof of Theorem 1

2.1 Definition of Sub-Gaussian and Sub-Exponential

First we give typical definitions of sub-Gaussian random variable and random vector, meanwhile, definition of sub-Exponential random variable is also provided.

Definition 3

(sub-Gaussian random variable) A random variable $X\in\mathbb{R}$ is said to be sub-Gaussian with variance proxy $\sigma_{2}$ if $\mathbb{E}[X]=0$ and its moment generating function satisfies

[TABLE]

In this case we write $X\sim\texttt{subG}(\sigma^{2})$.

Definition 4

(sub-Gaussian random vector) A random vector $\mathbf{x}\in\mathbb{R}^{d}=(x_{1},\cdots,x_{d})$ is called sub-Gaussian with variance proxy $\sigma^{2}$ if all its coordinates are sub-Gaussian random variables with variance proxy $\sigma^{2}$.

Definition 5

(sub-Exponential random variable) A random variable $X\in\mathbb{R}$ is said to be sub-Exponential with parameter $\lambda$ if $\mathbb{E}[X]=0$ and its moment generating function satisfies

[TABLE]

In this case we write $X\sim\texttt{subE}(\lambda)$.

**Remark. ** Attention that definitions above all require a zero-mean constraint, which is not necessary in analysis ”light tail” property. Hence, for random variable that is not zero-mean but satisfies condition (14) is called generalized sub-Gaussian. And definitions for generalized sub-Gaussian random vector and generalized sub-Exponential random variable are similar.

2.2 Proof of Theorem 1

Theorem 1 in the paper presents a vector concentration inequality is shown for sub-Gaussian random vector sequence in the following which plays an important role in proving Lemma 2 and Lemma 3 in our paper. To prove Theorem 1 in the paper, first, we present following Lemma 5 to show that the norm of a sub-Gaussian random vector is a generalized sub-Gaussian random variable.

Lemma 5

If $\mathbf{x}\in\mathbb{R}^{d}$ is a sub-Gaussian random vector with variance proxy $\sigma^{2}$, then

[TABLE]

Proof.

The conclusion here is a direct corollary from Theorem 3.1 in BP (10), where $\mathbf{c}=\mathbf{1}=(1,\cdots,1)$, $G=\lVert\cdot\rVert$, and $B(\mathbf{c},\mathbf{x})=\sum_{k=1}^{d}c_{k}\tau(x_{k})=d\sigma$.

Then we present the following Lemma 6 to show the equivalence between sub-Gaussian random variable and sub-Exponential variable, which plays an important role in proving Theorem 1.

Lemma 6

Let $X$ be a sub-Gaussian random variable, i.e., $X\sim\texttt{subG}(\sigma^{2})$. Then the random variable $Z=X^{2}-\mathbb{E}[X^{2}]$ is sub-Exponential: $Z\sim\texttt{subE}(16\sigma^{2})$.

Proof.

We prove this lemma by definition.

[TABLE]

where (17) and (18) because of Jensen’s Inequality. (2.2) holds because of $\mathbb{E}[\lVert X\rVert^{k}]\leq(2\sigma^{2})^{k/2}k\Gamma(k/2)$ in Ver (10). The last step (20) holds because the condition in definition of sub-Exponential, i.e., $|s|\leq 1/(16\sigma^{2})$.

Remark. Attention that in the proof of Lemma 6, we didn’t use the zero mean property of sub-Gaussian random variable $X$. Hence, Lemma 6 can be applied to a generalized sub-Gaussian random variable without requiring zero-mean condition.

Now, let’s begin prove the Lemma 2 in the paper stated as follows,

For a sub-Gaussian random vector sequence $\{\mathbf{x}(t)\}$ with a variance proxy sequence $\{\sigma_{t}\}$, there exists a corresponding positive bounding sequence $\{\gamma_{t}\}$, such that

[TABLE]

Proof.

Consider any vector in the sequence, say $\mathbf{x}(t)$. Because it is a sub-Gaussian random vector, directly applying Lemma 5, we have

[TABLE]

which means for a sub-Gaussian random vector, its norm is a generalized sub-Gaussian random variable. Here, ”generalized” means it may not meet the zero-mean condition.

Because we didn’t use the zero mean property of sub-Gaussian random variable in the proof of Lemma 6, it can also be applied to generalized sub-Gaussian random variable. Let $Z=\lVert\mathbf{x}(t)\rVert^{2}-\mathbb{E}[\lVert\mathbf{x}(t)\rVert^{2}]$, then $Z\sim\texttt{subE}(16\sigma_{t}^{2})$, specifically,

[TABLE]

Thus, for $\forall|s|\leq 1/(4\sigma_{t})$, we have

[TABLE]

Obviously, we can choose a sufficient small positive constant $s$ as $\gamma_{t}$, such that

[TABLE]

And $\{\gamma_{t}\}$ is exactly the bounding sequence as we desired. This completes the proof.

Now, we prove the Theorem 1 in the paper stated as follows,

Theorem 3

In an Euclidean space $(\mathbb{R}^{n},\lVert\cdot\rVert_{2})$, let E-valued martingale-difference sub-Gaussian sequence $\xi^{\infty}$ with a corresponding bounding sequence $\sigma^{N}=[\sigma_{1};\cdots;\sigma_{N}]$. Let $S_{N}=\sum_{i=1}^{N}\xi_{i}$, then for all $N\geq 1$ and $\gamma\geq 0$:

[TABLE]

Proof.

First of all, for a sub-Gaussian sequence $\xi^{\infty}$ with corresponding bounding sequence $\sigma^{\infty}$, from Theorem 1 in paper, we have $\mathbb{E}_{i-1}\left\{\exp\{\frac{\lVert\xi_{i}\rVert^{2}}{\sigma_{i}^{2}}\}\right\}=\exp\{1\}$. Besides, an Euclidean space $(\mathbb{R}^{n},\lVert\cdot\rVert_{2})$ is 1-smooth and 1-regular, which means $\kappa=1$. Then, it follows immediately according to Theorem 2.1 proposed in JN (08).

3 Proof of Theorem 2

Firstly, the following generalized summation by parts is essential for the proof of main theorem in our paper.

Lemma 7

Let $\left\langle{f_{n}}\right\rangle$ and $\left\langle{g_{n}}\right\rangle$ be two sequences. And denote $G_{n}=\sum_{k=1}^{n}g_{k}$, then for $m\geq 2$ we have,

[TABLE]

Proof.

The right-hand side can be easily verified by expanding $g_{k}$ in the left-hand as $G_{k}-G_{k-1}$.

Remark. When $m=1$, the same derivation gives the famous Abel transformation,

[TABLE]

3.1 Proof of Lemma 2 in paper

Lemma 2 in our paper states as follows,

Let $\{\mathbf{x}(t)\}$ be an $E$-valued martingale-difference $d$-dimension sub-Gaussian random vector sequence, with corresponding bounding sequence $\{\gamma_{t}\}$, and $Z(t)\in\mathbb{R}^{d\times d},Y(t)=(1-\mu)Y(t-1)+\mu Z(t),t\geq 1$. Then for $\mu\in(0,1)$, with a probability at least $1-\frac{\delta}{2}$, we have

[TABLE]

where $Z^{*}=\sup_{k=1,\cdots,t}\lVert Z(k)\rVert$ and $\gamma^{*}=\sup_{k=1,\cdots,t}\gamma_{k}$.

Proof.

Denote $S(t,k)\triangleq\sum_{i=k}^{t}\mathbf{x}(i)$, then with the recursive property of $Y(t)$, left side can be expanded by Lemma 7,

[TABLE]

And it could be separated into two parts, i.e.,

[TABLE]

For $k=1,\cdots,t$, based on Lemma 3, we have

[TABLE]

taking by the union bound over $t>0$, and let $\gamma^{*}=\sup_{k=1,\cdots,t}\gamma_{k}$ the following holds in a probability at least $1-\frac{\delta}{2}$,

[TABLE]

Conditioning on all above concentration inequalities hold, for $(a)$, we have

[TABLE]

Hence, combining $(a)$ and $(b)$, we complete the proof. ∎

3.2 Proof of Lemma 3 in paper

Lemma 3 in our paper states as follows,

Let $\{\epsilon(t)\}$ be an independent (or $E$-valued martingale-difference) sub-Gaussian random variable sequence, with corresponding bounding sequence (i.e., variance proxy sequence) $\{\sigma_{t}\}$, and $\mathbf{x}(t)\in\mathbb{R}^{d},t\geq 1$. Then for $\mu\in(0,1)$, with a probability at least $1-\frac{\delta}{2}$, we have

[TABLE]

where $\sigma^{*}=\sup_{k=1,\cdots,t}\lVert\mathbf{x}(k)\rVert\cdot\sup_{k=1,\cdots,t}\sigma_{k}$.

Proof.

First, we examine the ”light tail” condition, i.e.,condition $(\mathcal{C}_{\alpha}[\sigma^{\infty}])$ in Theorem 2.1 proposed in JN (08), let $\sigma^{\prime 2}_{i}=(\max\lVert\mathbf{x}(k)\rVert^{2})\sigma^{2}_{i},\forall i\geq 1$, then

[TABLE]

Hence, conclusion in Lemma 3 holds on sequence $\{\mathbf{x}(t)\epsilon(t)\}$. Similar to the proof in proof of Lemma 2(original paper), by taking $Y(t)\equiv Z(t)\equiv I$, we complete the proof. ∎

3.3 Proof of Theorem 2

Now let’s begin to prove the main theorem, i.e., Theorem 2 in our paper states as follows.

Theorem 4

Assume following conditions be satisfied:

• (i)

drift term $\{\mathbf{s}(t)\}$ is an $E$-valued martingale-difference sub-Gaussian random vector sequence, with corresponding bounding sequence $\{\gamma_{t}\}$;

• (ii)

output noise $\{\epsilon(t)\}$ is an independent (or $E$-valued martingale-difference) sub-Gaussian random variable sequence, with corresponding bounding sequence (i.e., variance proxy sequence) $\{\sigma_{t}\}$.

Then with a probability at least $1-\delta$, we have

[TABLE]

where $K=\sup_{k=1,\cdots,t}\lVert P(k)\rVert$, $x^{*}=\sup_{k=1,\cdots,t}\lVert\mathbf{x}(k)\rVert$, $\sigma^{*}=\sup_{k=1,\cdots,t}\lVert\mathbf{x}(k)\rVert\cdot\sup_{k=1,\cdots,t}\sigma_{k}$ and $\gamma^{*}=\sup_{k=1,\cdots,t}\gamma_{k}$.

Proof.

Define $R(t)\triangleq P^{-1}(t)$, then we have

[TABLE]

so we only need to consider $\|R(t)\tilde{\mathbf{w}}(t)\|$. Recall model assumption, drifting assumption and update rule,

[TABLE]

we can obtain linear recurrence relations of $R(t)\tilde{\mathbf{w}}(t)$,

[TABLE]

which gives

[TABLE]

Hence, by Minkowski’s inequality, it can be bounded by

[TABLE]

When output noise $\{\epsilon(t)\}$ is an independent (or $E$-valued martingale-difference) sub-Gaussian random variable sequence, with corresponding bounding sequence (i.e., variance proxy sequence) $\{\sigma_{t}\}$, then directly apply Lemma 3(original paper) on the second term in (24) gives, with at least $1-\frac{\delta}{2}$,

[TABLE]

where $\sigma^{*}=\sup_{k=1,\cdots,t}\lVert\mathbf{x}(k)\rVert\cdot\sup_{k=1,\cdots,t}\sigma_{k}$.

When the drift term $\{\mathbf{s}(t)\}$ is an $E$-valued martingale-difference sub-Gaussian random vector sequence, with corresponding bounding sequence $\{\gamma_{t}\}$. Besides, it’s trivial to verified that $R(k)$ meets the decreasing recursive structure stated, then directly apply Lemma 2 on the last term in (24) gives, with at least $1-\frac{\delta}{2}$,

[TABLE]

where $x^{*}=\sup_{k=1,\cdots,t}\lVert\mathbf{x}(k)\rVert$ and $\gamma^{*}=\sup_{k=1,\cdots,t}\gamma_{k}$.

Combining (25) and (26) by union bound, we get desired estimate error bound in assertion, hence complete the proof.∎

4 Additional Experiments

4.1 Regression

In this part, we compare the proposed DFOP to state-of-the-art streaming regression approaches and on both synthesis and real-world datasets. The comparison methods are (a) RLS, least square approach solved in a recursive manner, (b) Online Bagging (OB) OR (01), (c) AddExp.C KM (05), (d) EOS-ELM LSH (09), (e) $\text{Learn}^{++}$.NSE EP (11), and (f) OAUE BS (14), where (b)-(f) are all ensemble style approaches which dynamically adapt the models based on the coming data item or data batch, the differences are the strategies on how to update models like adding new models, excluding models or adjusting models’ weights.

Both synthesis dataset hyperplane and real-world datasets Sulfur recovery unit and Debutanizer column are employed to demonstrate the effectiveness of proposed DFOP. The detailed descriptions of datasets are included in Section 4.1 of supplementary material.

The performance of various streaming regression approaches is assessed by MSE (mean square error) between ground-truth and predict values over 10 trails. Both mean and standard deviation are reported in Table 3. It has to be pointed out that, not all comparisons are fair. DFOP is a one-pass approach which needs to scan each data item only once. For most ensemble style methods, data items are usually scanned many times for the need of dynamically update ensemble models. Besides, $\text{Learn}^{++}$.NSE is batch basis, nevertheless, DFOP is incremental and could update the model by a single data item.

From Table 3, we can see that DFOP outperforms RLS, and achieved a satisfying performance with lowest MSE in three datasets. Meanwhile, since the real drifting property of synthesis dataset is clear, we also present the trend chart of DFOP with different forgetting factors in terms of square loss over generated test data and estimate error $\lVert\tilde{\mathbf{w}}(t)\rVert=\lVert\mathbf{w}(t)-\hat{\mathbf{w}}(t)\rVert$ in Figure 3. And we can see that the error of DFOP is significantly lower than RLS, which is a special case of DFOP when forgetting factor is 0. This proves the effectiveness of forgetting mechanism. Besides, in the right side, we can see that $\lVert\tilde{\mathbf{w}}(t)\rVert$ is tending towards stability as time series go to the infinity which is consistent with our theory proposed in the original paper even the dataset does not exactly meet the assumptions proposed in Theorem 2.

4.2 Robustness Comparisons

Additionally, the robustness of all these different algorithms are compared. Concretely speaking, for a particular algorithm algo, similar to definition in VDG*+* (02), the robustness here is defined as the proportion between its accuracy and the smallest accuracy among all compared algorithms,

[TABLE]

Apparently, the worst algorithm has $r_{algo}=1$, and the others have $r_{algo}\geq 1$, the greater the better. Hence, the sum of $r_{algo}$ over all datasets indicates the robustness of for algorithm $algo$. The greater the value of the sum, the better the performance of the algorithm.

We provide a robustness comparison on six compared algorithms and DFOP over 20 datasets in Figure 4. From the figure, we can see that DFOP achieves the best over 20 datasets, and RLS ranks last as expected since it didn’t consider the evolving distribution in datasets at all. For sliding window type approaches, i.e., SVM-win and 1NN-win, their performances are not quite satisfying, even if we have chosen a relatively optimal window size by cross-validation with different data splits. For two batch type approaches, i.e., SVM-fix and SVM-ada. Performance of SVM-fix is comparable to DFOP, however, it’s curious that SVM-ada demonstrates a frustrating performance, even worse than SVM-fix. This might because the estimate generalization error is not consistent with empirical one. DWM shows a competitive performance, nevertheless, it needs to scan data many times to maintain a dynamic expert pool. As the only one-pass learning algorithm above except for RLS, DFOP shows a satisfying performance and property in handling data stream with distribution change.

5 Detailed Descriptions of Datasets

5.1 Regression Datasets

hyperplane KM (05) is a benchmark synthesis dataset for regression scenario, where feature consists of 10 variables, each is uniform sampled from $[0,1]$. There are in total 2,000 data samples with four stages, target concepts are $(x_{i}+x_{i+1}+x_{i+2})/3>0.5$, with $i=1,2,4,7$ in each stage.

• -

Sulfur recovery unit FGRX (07) is, a real-world dataset which is a record of gas diffusion. Feature consists of 5 different chemical and physical indexes, with in total 10,081 data samples. There are two outputs in original dataset represent the concentration of $\text{SO}_{2}$ and $\text{H}_{2}\text{S}$, and we split it into SRU-1 and SRU-2, respectively.

• -

Debutanizer column FGRX (07) is, a real-world dataset which is a record of chemical reactions, with in total 2,394 data samples consisting of 7 different features. The output represents C4 content in the debutanizer bottoms.

5.2 Classification Datasets

Chess Zli (11) is constructed by the data from chess.com portal, which consists of game records of players over a period from December 2007 to March 2010, a total of 533 instances with 7 attributes. The label indicates win or loss in the game.

• -

Usenet KTV (08) is split into Usenet-1 and Usenet-2 which both consist of 1,500 instances with 100 attributes based on 20 newsgroups collection. They simulate a stream of messages from different newsgroups that are sequentially presented to a user, who then labels them according to his/her personal interests.

• -

Luxembourg Zli (11) is constructed using European Social Survey data. There are 1,900 instance with 32 attributes in total, each instance is an individual and attributes are formed from answers to the survey questionnaire. The label indicates high or low internet usage.

• -

Spam Detection KTB*+* (09), real-world textual dataset that use email messages from the Spam Assassin Collection, and boolean bag-of-words approach is adopted to represent emails. It consists of 9,324 instances with 500 attributes, and label indicates spam or legitimate.

• -

Weather EP (11) Weather dataset is originally collected from the Offutt Air Force Base in Bellevue, Nebraska. 18,159 instances are presented with an extensive range of 50 years ($1949-1999$) and diverse weather patterns. Eight features were selected based on their availability, eliminating those with a missing feature rate above 15%. The remaining missing values were imputed by the mean of features in the preceding and following instances. Class labels are based on the binary indicator(s) provided for each daily reading of rain with 18,159 daily readings: 5698 (31%) positive (rain) and 12,461 (69%) negative (no rain).

• -

Powersupply CKH*+* (15) contains three year power supply records including 29,928 instances with 2 attributes from 1995 to 1998, and our learning task is to predict which hour the current power supply belongs to. We relabeled into binary classification according to p.m. or a.m.

• -

Electricity HW (99) is wildly adopted and is collected from the Australian New South Wales Electricity Market where prices are affected by demand and supply of the market. The dataset contains 45,312 instances with 8 features. The class label identifies the change of the price relative to a moving average of the last 24 hours.

The basic information for all datasets mentioned in the original paper is summarized in the following Table 4.