Space-efficient estimation of empirical tail dependence coefficients for bivariate data streams

TL;DR

This paper introduces a space-efficient method to approximate empirical tail dependence coefficients in bivariate data streams, maintaining error bounds regardless of stream length, with theoretical and practical validation.

Contribution

It develops a novel, space-efficient approximation for bivariate empirical copulas that accurately models tail dependence with stream-length invariant error bounds.

Findings

Accurately approximates tail dependence in data streams

Error bounds remain invariant with stream length

Validated with real-world netflow data

Abstract

This article proposes a space-efficient approximation to empirical tail dependence coefficients of an indefinite bivariate stream of data. The approximation, which has stream-length invariant error bounds, utilises recent work on the development of a summary for bivariate empirical copula functions. The work in this paper accurately approximates a bivariate empirical copula in the tails of each marginal distribution, therefore modelling the tail dependence between the two variables observed in the data stream. Copulas evaluated at these marginal tails can be used to estimate the tail dependence coefficients. Modifications to the space-efficient bivariate copula approximation, presented in this paper, allow the error of approximations to the tail dependence coefficients to remain stream-length invariant. Theoretical and numerical evidence of this, including a case-study using the Los Alamos National Laboratory netflow data-set, is provided within this article.

Figures

Equations

$$C(u_1,u_2)=F_{X_{(1)},X_{(2)}}\left(F_{X_{(1)}}^{-1}(u_1),F_{X_{(2)}}^{-1}(u_2)\right),$$

$$\underset{x_{(1)}\in \mathbb{R}}{\inf }{F}_{X_{(1)}}(x_{(1)})\ge u_1\text{and}\underset{x_{(2)}\in \mathbb{R}}{\inf }{F}_{X_{(2)}}(x_{(2)})\ge u_2,$$

$$\hat{C}(u_1,u_2)=\frac{1}{n}\sum _{i=1}^n\left(1_{x_{(1)}^i\le {\tilde{x}}_{(1)}^{\lceil u_{1}n\rceil }}\right)\left(1_{x_{(2)}^i\le {\tilde{x}}_{(2)}^{\lceil u_{2}n\rceil }}\right).$$

$$\hat{C}(u_1,u_2)=\frac{n_1}{n}\sum _{i\in I}{1}_{x_{(2)}^i\le {\tilde{x}}_{(2)}^{\lceil u_{2}n\rceil }},$$

$${\hat{F}}_{n,(2)}^{-1}(u_2)={\tilde{x}}_{(2)}^{\lceil u_{2}n\rceil },$$

$${\hat{F}}_{n_1,(2)}(x):=\frac{1}{n_1}\sum _{i=1}^{n_1}{1}_{x_{(1)}^{I(i)}\le x},$$

$$\hat{C}(u_1,u_2)=\frac{n_1}{n}{\hat{F}}_{n_1,(2)}\left({\hat{F}}_{n,(2)}^{-1}(u_2)\right).$$

$${\lambda }_L=\underset{u\to 0}{\lim }\frac{C(u,u)}{u},$$

$${\lambda }_U=\underset{u\to 1}{\lim }\frac{1-2u+C(u,u)}{(1-u)}.$$

$${\hat{\lambda }}_L=\underset{i\to 0}{\lim }\frac{\hat{C}(i/n,i/n)}{i/n},$$

$${\hat{\lambda }}_U=\underset{i\to n}{\lim }\frac{1-\frac{2i}{n}+\hat{C}(i/n,i/n)}{1-\frac{i}{n}}.$$

$${\hat{\lambda }}_L(i/n)=\frac{\hat{C}(i/n,i/n)}{i/n},$$

$${\hat{\lambda }}_U(i/n)=\frac{1-\frac{2i}{n}+\hat{C}(i/n,i/n)}{1-\frac{i}{n}}.$$

$$\left|{\hat{\lambda }}_L(i/n)-\frac{\tilde{C}(i/n,i/n)}{i/n}\right|=\frac{n}{i}\left|\hat{C}(i/n,i/n)-\tilde{C}(i/n,i/n)\right|\le \frac{5ϵn}{i},$$

$$\left|{\hat{\lambda }}_U(i/n)-\frac{1-\frac{2i}{n}+\tilde{C}(i/n,i/n)}{1-(i/n)}\right|=\frac{n}{j}\left|\hat{C}(i/n,i/n)-\tilde{C}(i/n,i/n)\right|\le \frac{5ϵn}{j}.$$

$$j\in [\lceil un\rceil -ϵ\min (u,(1-u))n,\lceil un\rceil +ϵ\min (u,(1-u))n].$$

$$g^i=r_{min,Q}(v_i)-r_{min,Q}(v_{i-1}),$$

$$r_{min,Q}(v_i)=\sum _{j=1}^i{g}^j,r_{max,Q}(v_i)=r_{min,Q}(v_i)+{\Delta }^i.$$

$$r_{max,Q}(v_{i+1})-r_{min,Q}(v_i)\le 2ϵ\min \left(r_{min,Q}(v_i),n-r_{max,Q}(v_{i+1})\right).$$

$$\text{arg}\min_{j}\Bigg{\{}\sum^{i}_{k=j}g_{(1)}^{k}+\Delta_{(1)}^{i}\leq 2\epsilon\min\left(r_{min,S_{(1)}}(v_{j-1}),n-r_{max,S_{(1)}}(v_{i})\right)=2\epsilon\min\left(\sum^{j-1}_{k=1}g_{(1)}^{k},n-\sum^{i}_{k=1}g_{(1)}^{k}-\Delta_{(1)}^{i}\right)\Bigg{\}}.$$

$${\hat{n}}_1=\sum _{i=1}^E{g}_{(1)}^i=\sum _{i=1}^E\sum _{j=1}^{L_i}{g}_{(2)}^{i,j}.$$

$${\hat{F}}_{{\hat{n}}_1,(2)}(y)=\frac{1}{{\hat{n}}_1}\sum _{i=1}^{{\hat{n}}_1}{1}_{{\tilde{x}}_{(2)}^{\hat{I}(i)}\le y},$$

$$\widetilde{\stackrel{~}{C}}(u_1,u_2)=\frac{{\hat{n}}_1}{n}{\widetilde{\stackrel{~}{F}}}_{{\hat{n}}_1,(2)}\left({\widetilde{\stackrel{~}{F}}}_{n,(2)}^{-1}(u_2)\right).$$

$$r_{max,Q}(v_{i+1})-r_{min,Q}(v_i)\le 2ϵ\min \left(r_{min,Q}(v_i),n-r_{max,Q}(v_{i+1})\right).$$

$$r_{min,M(Q_1,Q_2)}(z_k)=\{\begin{array}{ll}{r}_{min,Q_2}(w_1)+r_{min,Q_1}(z_k),& \text{if }{w}_1\text{ exists}\\ r_{min,Q_1}(z_k),& \text{otherwise},\end{array}$$

$$r_{max,M(Q_1,Q_2)}(z_k)=\{\begin{array}{ll}{r}_{max,Q_2}(w_2)+r_{max,Q_1}(z_k)-1,& \text{if }{w}_2\text{ exists}\\ r_{max,Q_2}(w_1)+r_{max,Q_1}(z_k),& \text{otherwise}.\end{array}$$

$$r_{max,M(Q_1,Q_2)}(z_{k+1})-r_{min,M(Q_1,Q_2)}(z_k)\le 2ϵ\min \left(r_{min,M(Q_1,Q_2)}(z_k),n-r_{max,M(Q_1,Q_2)}(z_{k+1})\right).$$

$$\begin{split}r_{max,M(Q_{1},Q_{2})}(z_{k+1})-r_{min,M(Q_{1},Q_{2})}(z_{k})&\leq(r_{max,Q_{1}}(x_{2})-r_{min,Q_{1}}(x_{1}))+(r_{max,Q_{2}}(w_{2})-r_{min,Q_{2}}(w_{1})-1)\\ \quad&\leq 2\epsilon\big{(}\min\left(r_{min,Q_{1}}(x_{1}),n_{1}-r_{max,Q_{1}}(x_{2})\right)+\ldots\\ \quad&\ldots\min\left(r_{min,Q_{2}}(w_{1}),n_{2}-r_{max,Q_{2}}(w_{2})\right)\big{)}\\ \quad&\leq 2\epsilon\min\left(r_{min,M(Q_{1},Q_{2})}(z_{k}),(n_{1}+n_{2})-r_{max,M(Q_{1},Q_{2})}(z_{k+1})\right).\end{split}$$

$$\begin{array}{rl}{r}_{max,M(Q_1,Q_2)}(z_{k+1})-r_{min,M(Q_1,Q_2)}(z_k)& \le (r_{max,Q_1}(x_2)-r_{min,Q_1}(x_1))\le 2ϵ\min \left(r_{min,Q_1}(x_1),n_1-r_{max,Q_1}(x_2)\right)\\ & \le 2ϵ\min \left(r_{min,M(Q_1,Q_2)}(z_k),(n_1+n_2)-r_{max,M(Q_1,Q_2)}(z_{k+1})\right),\end{array}$$

$$\begin{split}r_{max,M(Q_{1},Q_{2})}(z_{k+1})-r_{min,M(Q_{1},Q_{2})}(z_{k})&\leq(r_{max,Q_{2}}(w_{1})+r_{max,Q_{1}}(x_{2}))-(r_{min,Q_{2}}(w_{1}))+r_{min,Q_{1}}(x_{1}))\\ \quad&\leq 2\epsilon\big{(}\min\left(r_{min,Q_{1}}(x_{1}),n_{1}-r_{max,Q_{1}}(x_{2})\right)+\ldots\\ \quad&\min\left(r_{min,Q_{2}}(w_{1}),n_{2}-r_{max,Q_{2}}(w_{1})\right)\big{)}\\ \quad&\leq 2\epsilon\min\left(r_{min,M(Q_{1},Q_{2})}(z_{k}),(n_{1}+n_{2})-r_{max,M(Q_{1},Q_{2})}(z_{k+1})\right).\end{split}$$

Full text

\jnlcitation\cname

, and (\cyear2019), \ctitleSpace-efficient estimation of empirical tail dependence coefficients for bivariate data streams

††articletype: Article Type

1 Introduction

Streaming data arises in a wide array of contemporary research fields, including but not limited to the Internet of Things (IoT), cyber-security 2 and continuous sensor observation 12. The data is acquired continuously and usually at a fast pace. Typically one can’t store all of the data ever streamed due to memory constraints and it is infeasible to repeat statistical analyzes on the entire stream as it grows indefinitely over time. To deal with this problematic situation, estimation methods for many different statistical analyzes on such data streams have been proposed 11, 3. Some of these techniques use statistical summaries, where only a succinct number of carefully selected elements that have entered into the data stream are stored. One of the most common analyzes that the literature has focused on is quantile estimation for a univariate variable observed in a data stream (including median estimation) 4. Recently, 15 proposed an algorithm to generate an approximation to the bivariate empirical copula function (a common method of nonparametric dependence modelling) using a succinct statistical summary; this approximation has a guaranteed error bound.

This paper builds on the work in 15 and considers an important use of copulas: computing tail dependence coefficients between random variables 23. Tail dependence coefficients between random variables quantify their co-movement in the tails of the marginals. For example, two random variables may be weakly dependent in the vast majority of their probability space, however in the tails of this space they may be highly dependent. This behaviour is often seen in financial analysis 21, where sometimes two assets exhibit sharp price increases and decreases at similar times, but tend to have relatively uncorrelated typical daily price movements. The importance of tail dependence in fields such as hydrology 19 and energy 20 has also been studied. For the purpose of approximating empirical tail dependence coefficients between streams of data, with stream-length invariant error, the type of summary that was proposed in 15 is not sufficient since it returned uniform error over the marginal distributions. Therefore the copula summary in 15 results in an approximation to the tail dependence coefficient with linearly growing error w.r.t. the number of elements in the data stream. To remedy this, we propose to use relative accuracy quantile summaries from the univariate literature 7 within the copula summary, which allows suitable properties of the modified summaries’ approximation error to be proved. These properties lead to approximations of the tail dependence coefficients that have constant error w.r.t. the number of elements in the data stream.

This article is structured as follows. The next section introduces the empirical copula and how it can be used to construct empirical tail dependence coefficients between random variables observed through data. This section also describes the challenges associated with computing empirical copula approximations when the data is streamed sequentially, and how the copula summary proposed in 15 can be used to provide an approximation to copulas in this streaming regime. Then Sec. 3 introduces how one can adapt the summary to compute accurate approximations to empirical tail dependence coefficients for streaming data. Sections 4 and 5 provide a theoretical and numerical analysis of the approximation respectively. In particular, Sec. 5.3 provides a case-study of applying the proposed methodology to the Los Alamos National Laboratory (LANL) netflow data-set.

2 Bivariate empirical copulas and tail dependence

A copula is a dependence model between two or more different random variables 24. The work in 15 explains how one can consider higher dimensional (greater than two) copulas in the context of streaming data (the motivation of this paper), using decompositions involving pair-wise copula models 1. This is described further in Sec. 4.4. However, this aspect is outside the scope of the current study on tail dependence, and therefore for the remainder of this paper we will focus on the case where there is only two random variables. More specifically, a bivariate copula function $C(u_{1},u_{2})$, for $u_{1},u_{2}\in[0,1]$, is the joint distribution function between the random variables $X_{(1)}$ and $X_{(2)}$ where both marginals are uniformly distributed. The bivariate copula function is given by,

[TABLE]

where $F_{X_{(1)},X_{(2)}}(x_{(1)},x_{(2)})=P(X_{(1)}\leq x_{(1)},X_{(2)}\leq x_{(2)})$ is the joint cumulative distribution function (CDF) of $X_{(1)}$ and $X_{(2)}$, and $F^{-1}_{X_{(1)}}(u_{1})$ and $F^{-1}_{X_{(2)}}(u_{2})$ are the marginal generalized inverse CDFs (quantile functions). They are defined by,

[TABLE]

respectively 6. Let \big{\{}x_{(1)}^{i}\big{\}}_{i=1}^{n} and \big{\{}x_{(2)}^{i}\big{\}}_{i=1}^{n} be realisations (the data) of the random variables $X_{(1)}$ and $X_{(2)}$ respectively. In this case, a nonparametric empirical copula function is typically found to represent the dependence between the two data-sets in \big{\{}x_{(1)}^{i},x_{(2)}^{i}\big{\}}_{i=1}^{n}. The empirical copula 8 converges to the true copula function in (1), within the limit of $n\to\infty$. It is defined by,

[TABLE]

where $\left\lceil z\right\rceil=\text{arg}\min_{m\in\mathbb{Z}}(m\geq z)$ , $1$ is the indicator function and $\tilde{x}_{(k)}^{i}$ is the $i$’th order statistic of \big{\{}x_{(k)}^{i}\big{\}}_{i=1}^{n}, for $k=1,2$. Here, the mean over the products of indicator functions represents the approximation to the joint CDF of $X_{(1)}$ and $X_{(2)}$ in (1), whilst the ranked order statistics $\tilde{x}_{(k)}^{\left\lceil u_{k}n\right\rceil}$ within each indicator function represents the approximation to the marginal generalized inverse CDFs in (1). Since each term in the summation in (2) will only be non-zero if $x_{(k)}^{i}\leq\tilde{x}_{(k)}^{\left\lceil u_{k}n\right\rceil}$, for both $k=1,2$, then it suffices to sum the indicator functions corresponding to only one value of $k$ over all indices that satisfy $x_{(l)}^{i}\leq\tilde{x}_{(l)}^{\left\lceil u_{l}n\right\rceil}$ for $l=\{1,2\}\setminus k$. By doing this, the empirical copula can alternatively be written as 15,

[TABLE]

where $n_{1}$ is the cardinality of the set $I\subset[1,n]$, such that $x_{(1)}^{I}$ correspond to elements in \big{\{}x_{(1)}^{i}\big{\}}_{i=1}^{n} that satisfy $x_{(1)}^{I}\leq\tilde{x}_{(1)}^{\left\lceil u_{1}n\right\rceil}$. Note that $u_{1}$ is now accounted for via the set $I$. Now let $\hat{F}_{n,(2)}^{-1}(u_{2})$ be the empirical quantile function 18 of \big{\{}x_{(2)}^{i}\big{\}}_{i=1}^{n}, for which we will use the approximation,

[TABLE]

to the quantile function $F^{-1}_{X_{(2)}}(u_{2})$. Also let $\hat{F}_{n_{1},(2)}(x)$ be the empirical CDF given by,

[TABLE]

where $I(i)$ is the $i$’th element of $I$111Note that this expression is only taken over elements in the set $I$. Using this notation, another way of writing (3) is,

[TABLE]

For more information on the statistical explanation behind the approximation in (5), turn to 15.

One of the by-products of a copula is the computation of the tail dependence coefficients (there is an upper and a lower one) 23. These coefficients allow one to study the dependence in the tails of each of the marginals $X_{(1)}$ and $X_{(2)}$. For example, $X_{(1)}$ and $X_{(2)}$ may have low dependence over their entire probability space, however they could have very high dependence when both $X_{(1)}$ and $X_{(2)}$ take extreme values. This aspect of dependence is important in many applications, for example in financial analysis where it is crucial to realise if two assets have a high relative probability of both crashing at similar times. The lower tail dependence coefficient between $X_{(1)}$ and $X_{(2)}$ can be computed directly via the copula function,

[TABLE]

and so too the upper tail dependence coefficient,

[TABLE]

The focus of this paper is on the estimation of empirical tail dependence coefficients for streaming data; this is done by utilising nonparametric empirical copulas. Under true model assumptions, one can estimate tail dependence by fitting a suitable parametric copula $C(u_{1},u_{2})$ to the data and this could offer a better estimate of the tail dependence coefficients than using any nonparametric copula function. An example of such a parametric copula is the Gaussian copula, where Gaussianity of the data is assumed. However in practice, if the assumptions on the data stream are too restrictive, then it is suitable to consider a nonparametric estimate of the underlying copula function. Estimators of the tail dependence coefficients that utilise these nonparametric copula estimates $\hat{C}(u_{1},u_{2})$, that fall into the scope of this paper, can be used in this case. For a detailed account of these estimates, see 10 and 22. One could employ a model goodness-of-fit technique (e.g. Chi-Square), such as a test for Gaussianity in the case of considering a Gaussian copula, to confirm if this is the case. If it is more suitable to employ a parametric copula, the computational challenges associated with updating the tunable parameter estimates with indefinite data streams would have to be considered by the user, e.g. see 15 for more detail on this. On the other hand if a nonparametric copula estimate is a reasonable choice in a particular context, then the proposed methodology in this paper can be used to provide approximations of tail dependence coefficients over a data stream. This is the assumption that we make for the remainder of the paper.

Using the empirical copula $\hat{C}(u_{1},u_{2})$, one estimate of the empirical lower tail dependence coefficient is given by 5,

[TABLE]

and one estimate of the empirical upper tail dependence coefficient is given by,

[TABLE]

These are consistent with the tail dependence coefficients in (6) and (7) respectively as $n\to\infty$, since the empirical copula is also consistent 8. Empirically one cannot take this limit and therefore it suffices to study the following functions,

[TABLE]

and

[TABLE]

For $i=n-1,$ $n-2,\ldots,$ $2,$ $1$, the functions in (10) and (11) describe the path of $\hat{\lambda}_{L}$ and $\hat{\lambda}_{U}$ as $i$ tend to 1 and $n$ respectively 5. Note that when evaluating these functions with a fixed value of $i$, they are consistent with the limits in (8) and (9) as $n\to\infty$, since $\lim_{n\to\infty}(i/n)=0$. It has been proposed to evaluate the functions in (10) and (11) with the minimum and maximum values of $i$ that the functions are decreasing and increasing for respectively 5. However for the scope of this paper, which will estimate these functions for an arbitrary fixed value of $i$, this particular selection is not justified further. This paper will instead concentrate on the approximation of the functions in (10) and (11) when the empirical copula function $\hat{C}(u_{1},u_{2})$ must be constructed over a data stream. The next section will propose an approximation to the tail dependence functions through approximation to the empirical copula, formed via a succinct summary of the data stream.

2.1 Bivariate copula summaries for streaming data

Streaming data is the scenario in which say, the bivariate data stream \big{\{}x_{(1)}^{i},x_{(2)}^{i}\big{\}}_{i=1}^{n}, is added to sequentially over (possibly indefinite) time. In the context of streaming data, it is not possible to store all of the data points in the stream or be able to consistently re-compute the order statistics for the empirical copula function (as mentioned above). This is typically due to restrictions on runtime and/or memory/storage. Quantile summaries are a common way of maintaining an approximation to the empirical quantile function in (4) as an univariate data stream is added to, whilst only storing a succinct number of elements from the stream in space-memory 13. On this note, define an $\epsilon$-approximate quantile summary $Q$, to be an approximation to the quantile function $\hat{F}_{n,(k)}^{-1}(u)=\tilde{x}_{(k)}^{\left\lceil un\right\rceil}$, that returns a value $\tilde{x}_{(k)}^{j}$, for $k=1,2$, where $j\in[\left\lceil un\right\rceil-\epsilon n,\left\lceil un\right\rceil+\epsilon n]$. An algorithm to construct such an summary was given in 13.

The work in 15 proposed another summary made up of a series of different $\epsilon$-approximate quantile summaries. This copula summary maintained an approximation $\tilde{C}(u_{1},u_{2})$ to the bivariate empirical copula function in (5) over the data stream \big{\{}x_{(1)}^{i},x_{(2)}^{i}\big{\}}_{i=1}^{n}. It was shown that an approximation within $[\hat{C}(u_{1},u_{2})-5\epsilon,\hat{C}(u_{1},u_{2})+5\epsilon]$ can be achieved. Just like the univariate quantile summaries that it is composed from, the copula summary was space-efficient and stored only a succinct number of elements from the data stream. The extension to this summary, proposed in Sec. 3.1, will allow an approximation to empirical tail dependence coefficients of data streams to be computed.

3 Estimation of tail dependence for data streams

The copula summary presented in 15 was not suitable to find coefficients of tail dependence for one main reason: the error of the approximation to (5) was uniform over a grid of evaluation points $u_{1}$ and $u_{2}$. Therefore the resolution of the approximation would be as refined in the tails of the two marginals as it would be for the medians of both marginals. This results in an approximation to the tail dependence coefficients (replacing $\hat{C}(i/n,i/n)$ with $\tilde{C}(i/n,i/n)$ respectively in (10) and (11)) that has error growing linearly with $n$. One can see this from the following error bound of the lower tail dependence coefficient function for fixed $i$,

[TABLE]

and the error bound of the upper tail dependence function for fixed $j=n-i$,

[TABLE]

Simply refining the prescribed error $\epsilon$ would be insufficient; one would need to sequentially refine $\epsilon$ as the stream gets longer, tending towards 0. The work presented in this paper is inspired by 7 in the univariate context, which considered biased quantile estimation and modified $\epsilon$-approximate quantile summaries to refine the error sufficiently at the tails, at the expense of error not at the tails. This will, as is apparent from the error analysis later in the paper (see Sec. 4), guarantee that the error from the approximations of the tail dependence coefficients stay fixed as the number of elements in the stream is increased.

3.1 Modifications to the copula summary

This section details the specific modifications to the $\epsilon$-approximate quantile summary introduced in Sec. 2.1, and therefore the copula summary, in order to obtain a suitable approximation to tail dependence coefficients. The proposed algorithm in 7 constructed a summary to maintain an approximation, with guaranteed error bounds, to the ‘biased’ quantiles $u^{l}$, for $l=1,$ 2, $\ldots$, and $u\in[0,1]$. An approximation to the biased quantiles should have error relative to the quantile query, such that the approximation to $u^{l}$ should have an error of $\pm\epsilon u^{l}$ (or if required, $\pm\epsilon(1-u^{l})$ by symmetry) rather than the uniform error of $\pm\epsilon$ from the standard quantile summary. This relative error allows one to refine the quantile approximation within the tails of an univariate distribution, and therefore is suited to the problem considered in this paper. We modify the construction of the summary proposed in 7 slightly to allow an error of $\pm\epsilon\min(u,1-u)$, assuming $l=1$, given that we would like to approximate both tails with relative error. On this note, define a $\epsilon u$-approximate quantile summary to be an approximation to $\hat{F}_{n,(k)}^{-1}(u)=\tilde{x}^{\lceil un\rceil}_{(k)}$, for $k=1,2$, which returns the value $\tilde{x}^{j}$, where

[TABLE]

Recall that the copula summary is composed of $L+1$ quantile summaries: \big{\{}S_{(1)}, $S_{(2)}^{1},\ldots,$ S_{(2)}^{L}\big{\}}. Proved later in Sec. 4, it will suffice to let the summaries \big{\{}S_{(1)}, $S_{(2)}^{1},\ldots,$ S_{(2)}^{L}\big{\}} be modified into $\epsilon u$-approximate summaries in order to obtain suitable approximations to the tail dependence coefficients. To modify them, the manner in which each summary is maintained and queried (through Sec. 3.2, 3.3 and 3.4) is changed from that explained in 15. First, the basic make-up of each quantile summary $Q$ remains the same from the initial summary proposed back in 13. Each summary is made up of tuples, Q=\big{\{}(v_{i},g^{i},\Delta^{i})\big{\}}_{i=1}^{L}. The values $v_{i}$, where $v_{i-1}\leq v_{i}<v_{i+1}$, are elements that have been seen in the data stream \big{\{}x_{(k)}^{i}\big{\}}_{i=1}^{n}, for $k=1,2$, so far. These values are maintained by the summary as ‘cover’ for a range of quantiles that one may wish to query. I.e. the value $v_{i}$ will be returned as an approximation to a nearby quantile. The values $g^{i}$ and $\Delta^{i}$ control the range of quantiles that the value $v_{i}$ is returned as an approximation to. They do this by governing the minimum, $r_{min,Q}(v_{i})$, and maximum, $r_{max,Q}(v_{i})$ rank that the value $v_{i}$ takes in the original data stream. We define

[TABLE]

and then,

[TABLE]

One also knows the length of the data stream at any one time via $n=\sum^{L}_{j=1}g^{j}$. In order to guarantee that the $\epsilon u$-approximate quantile summary maintains an approximation which satisfies (14), these minimum and maximum ranks must satisfy 7,

[TABLE]

See Appendix A for a proof of this.

In the copula summary, each of the subsummaries $S_{(2)}^{1}$, $S_{(2)}^{2},\ldots$, $S_{(2)}^{L}$ corresponds to an element inside of $S_{(1)}$ (therefore $S_{(1)}$ has the cardinality $L$). Whilst the summary $S_{(1)}$ contains elements (and information about their ranks) from the first component of the data stream, i.e. \big{\{}x_{(1)}^{i}\big{\}}_{i=1}^{n}, the summaries $S_{(2)}^{1}$, $S_{(2)}^{2},\ldots$, $S_{(2)}^{L}$ contain elements (and information about their ranks) from the second component of the data stream, i.e. \big{\{}x_{(2)}^{i}\big{\}}_{i=1}^{n}. On this note, let a tuple in $S_{(1)}$ be denoted by $(v_{i},g_{(1)}^{i},\Delta_{(1)}^{i})$ where v_{i}\in\big{\{}x_{(1)}^{i}\big{\}}_{i=1}^{n}, for $i=1,\ldots,L$, and let a tuple in $S_{(2)}^{i}$ be denoted by $(w_{i},g_{(2)}^{i,j},\Delta_{(2)}^{i,j})$ where w_{i}\in\big{\{}x_{(2)}^{i}\big{\}}_{i=1}^{n}, for $j=1,\ldots,L_{i}$ (therefore $S_{(2)}^{i}$ has the cardinality $L_{i}$). The parameters within these summaries are changed carefully over time as new elements are added to the data stream \big{\{}x_{(1)}^{i},x_{(2)}^{i}\big{\}}_{i=1}^{n} via the operations defined in the following three sections. It is important to remember that the proposed methodology here can only be used with data streams where $x_{(1)}^{i}$ and $x_{(2)}^{i}$ are acquired at the same frequency and the same time. This is because the operations described in the following sections take a pair of data points as an input. Considering the case where the data streams components are acquired at different rates is outside the scope of this paper; however this could be achieved by modifying the times at which the summary $S_{(1)}$ and the subsummaries $S_{(2)}^{1},S_{(2)}^{2},\ldots,S_{(2)}^{L}$ get updated using the operations described below.

3.2 Inserting an element into the copula summary

When the element $(x_{(1)}^{n+1},x_{(2)}^{n+1})$ enters the data stream, it should be inserted into the copula summary. This is done by inserting the tuple $(x_{(1)}^{n+1},1,\Delta_{(1)}^{*})$ into $S_{(1)}$. If $x_{(1)}^{n+1}<v_{1}$, then we insert the tuple at the start of $S_{(1)}$, and let $\Delta_{(1)}^{*}=1$. Conversely if $x_{(1)}^{n+1}\geq v_{L}$, then we insert the tuple at the end of $S_{(1)}$ and let $\Delta_{(1)}^{*}=1$. If $v_{i}\leq x_{(1)}^{n+1}<v_{i+1}$, then insert the tuple in between $v_{i}$ and $v_{i+1}$ and let $\Delta_{(1)}^{*}=\min(g_{(1)}^{i}+\Delta_{(1)}^{i}-1,g_{(1)}^{i+1}+\Delta_{(1)}^{i+1}-1)$. Now for the second component of the new element, let S_{(2)}^{*}=\big{\{}(x_{(2)}^{n+1},1,0)\big{\}} be a new quantile summary. This summary gets inserted between $S_{(2)}^{i}$ and $S_{(2)}^{i+1}$ in the copula summary \big{\{}S_{(1)},S_{(2)}^{1},\ldots,S_{(2)}^{L}\big{\}} if $v_{i}\leq x_{(1)}^{n+1}<v_{i+1}$, between $S_{(1)}$ and $S_{(2)}^{1}$ if $x_{(1)}^{n+1}<v_{1}$ or at the end of the copula summary if $x_{(1)}^{n+1}\geq v_{L}$. Finally, increase $L$ by 1.

3.3 Combining tuples in the copula summary

Combining the tuples in the summaries \big{\{}S_{(1)},S_{(2)}^{1},\ldots,S_{(2)}^{L}\big{\}} is occasionally required to remove unnecessary tuples from the summaries, whilst maintaining the elements required for the approximation to be of the desired accuracy. Providing that $L>3$, sequentially for each element $i\in[3,L-1]$ in $S_{(1)}$ we find the index $j\in[2,i]$ satisfying

[TABLE]

Once this value is found, the tuples $(v_{j},g_{(1)}^{j},\Delta_{(1)}^{j}),\ldots,$ $(v_{i},g_{(1)}^{i},\Delta_{(1)}^{i})$ can be combined into the new tuple $(v_{i},\sum^{i}_{k=j}g_{(1)}^{k},\Delta_{(1)}^{i})$. We use the condition on $j$ in (17) in order to guarantee (16) is satisfied. In addition to combining those tuples, we merge the tuples $S_{(2)}^{j},\ldots,$ $S_{(2)}^{i}$ into a new tuple $M(S_{(2)}^{j},\ldots,S_{(2)}^{i})$. See Sec. 4.1 for the implementation of this merging and the bound of the approximation error. Finally combine unnecesary tuples (e.g. $(w_{\cdot},g_{(2)}^{i,\cdot},\Delta_{(2)}^{i,\cdot})$) inside of this merged summary $M(S_{(2)}^{j},\ldots,S_{(2)}^{i})$, in the manner described earlier in this section. Insert this new summary in the place of $S_{(2)}^{j},\ldots,$ $S_{(2)}^{i}$ in the copula summary, such that the copula summary now is \big{\{}S_{(1)},S_{(2)}^{1},\ldots,S_{(2)}^{j-1},M(S_{(2)}^{j},\ldots,S_{(2)}^{i}),S_{(2)}^{i+1},\ldots\big{\}}.

3.4 Querying the copula summary

This section now describes how to query the copula summary (maintained over time using the operations in Sec. 3.2 and 3.3) for an approximation to $\hat{C}(u_{1},u_{2})$. We will denote this approximation by $\tilde{\tilde{C}}(u_{1},u_{2})$, as opposed to the approximation from the copula summary proposed in 15 composed of $\epsilon$-approximate quantile summaries, $\tilde{C}(u_{1},u_{2})$. First, we compute the approximation to the empirical quantile function $\hat{F}^{-1}_{n,(1)}(u_{1})=\tilde{x}_{(1)}^{\left\lceil u_{1}n\right\rceil}$ using the $\epsilon u$-approximate quantile summary $S_{(1)}$; denote this approximation by $\tilde{\tilde{F}}^{-1}_{n,(1)}(u_{1})$. Let $E$ be equal to the value of $i$ that satisfies $\tilde{\tilde{F}}_{n,(1)}^{-1}(u_{1})=v_{i}$, and find the total number of elements in the stream that have entered into the first $E$ subsummaries $S_{(2)}^{1},\ldots,$ $S_{(2)}^{E}$,

[TABLE]

Suppose that the indices of the $\hat{n}_{1}$ elements to have entered into the first $E$ subsummaries form the set $\hat{I}\subset[1,n]$; note this is an approximation to the set $I$ introduced in Sec. 2.

Next, let $M(S_{(2)}^{1},\ldots,S_{(2)}^{L})$ be a merged summary composed of all the subsummaries $S_{(2)}^{1},\ldots,$ $S_{(2)}^{L}$ (again for the implementation details of this merge, see Sec. 4.1). This summary can be queried for an approximation to $\hat{F}_{n,(2)}^{-1}(u_{2})$; denote this approximation by $\tilde{\tilde{F}}_{n,(2)}^{-1}(u_{2})$. Finally, let $M(S_{(2)}^{1},\ldots,S_{(2)}^{E})$ be a merged summary of the subsummaries $S_{(2)}^{1},\ldots,$ $S_{(2)}^{E}$. Then define the approximation $\tilde{\tilde{F}}_{\hat{n}_{1},(2)}(y)$ to the empirical CDF

[TABLE]

to be an inverse query of the summary $M(S_{(2)}^{1},\ldots,S_{(2)}^{E})$. The implementation details of this query, and a guarantee on it’s error with respect to the empirical CDF, is provided later in Sec. 4.2. In total, the copula summary approximation $\tilde{\tilde{C}}(u_{1},u_{2})$, to the empirical copula function $\hat{C}(u_{1},u_{2})$ is given by,

[TABLE]

4 Theoretical analysis of the approximation

This section provides a theoretical analysis of the approximation in (19) to the empirical copula function, and the resulting approximations to the empirical tail dependence coefficients. First, it is important to clarify error bounds for merged $\epsilon u$ - approximate quantile summaries, and an inverse query of a $\epsilon u$ - approximate quantile summary. Recall from (16) that the summary $Q$ is a $\epsilon u$-approximate quantile summary if two neighbouring elements $v_{i+1}$ and $v_{i}$ in $Q$ satisfy,

[TABLE]

The next two sections cover two preliminary bounds, before Sec. 4.3 outlines the error bound of the approximation to the tail dependence coefficients using the modified copula summary, proposed in this paper.

4.1 Merging $\epsilon u$-approximate quantile summaries

We recall from 14 that one can merge $\epsilon$-approximate quantile summaries $Q_{1}$ (length $L_{1}$) and $Q_{2}$ (length $L_{2}$) to obtain the quantile summary $M(Q_{1},Q_{2})$, containing the elements $Q_{1}\bigcup Q_{2}$, which is also $\epsilon$-approximate itself. It does this via the following method. Suppose $z_{k}$, for $k\in[1,L_{1}+L_{2}]$, is an element from $Q_{1}$ which exists in $M(Q_{1},Q_{2})$. If it exists, let $w_{1}$ be the largest element in $Q_{2}$ that is less than or equal to $z_{k}$. Also, if it exists, let $w_{2}$ be the smallest element in $Q_{2}$ that is greater than $z_{k}$. Then set,

[TABLE]

and

[TABLE]

It will now be proved that if $Q_{1}$ and $Q_{2}$ are $\epsilon u$-approximate then $M(Q_{1},Q_{2})$ is a $\epsilon u$-approximate summary as well. It is therefore necessary to show that

[TABLE]

Suppose $Q_{1}$ and $Q_{2}$ were constructed over data streams of length $n_{1}$ and $n_{2}$ respectively. For the case where $z_{k}$ and $z_{k+1}$ are from the same summary, let them equal $x_{1}$ and $x_{2}$ (say in the summary $Q_{1}$). If there exists both the elements $w_{1}$ (largest element in $Q_{2}$ that is less than or equal to $z_{k}$) and $w_{2}$ (smallest element in $Q_{2}$ that is greater than $z_{k+1}$) then we know that $w_{1}$ and $w_{2}$ are consecutive elements in $Q_{2}$. Thus,

[TABLE]

If $w_{1}$ doesn’t exist, then

[TABLE]

as $r_{max,Q_{2}}(w_{2})=1$ and if $w_{2}$ doesn’t exist, then

[TABLE]

For the case where $z_{k}$ and $z_{k+1}$ come from different summaries, say $z_{k}$ from $Q_{1}$ labelled by $x_{1}$, and $z_{k+1}$ from $Q_{2}$ labelled by $w_{2}$. Then w.l.o.g. let $x_{2}$ be the smallest element in $Q_{1}$ greater than $w_{2}$, and $w_{1}$ be the largest element in $Q_{2}$ less than or equal to $x_{1}$. Then,

[TABLE]

We now have the condition for a $\epsilon u$-approximate summary in (20) for all cases of membership to $Q_{1}$ and $Q_{2}$ for the elements $z_{k}$ and $z_{k+1}$. Therefore any $\epsilon u$-approximate summaries merged together will also be $\epsilon u$-approximate too.

4.2 Inversely querying $\epsilon u$-approximate quantile summaries

In this section, we would like to bound the approximation $\tilde{\tilde{F}}_{n,(k)}(x)$, for $x\in\mathbb{R}$ and $k=1,2$, to the empirical CDF $\hat{F}_{n,(k)}(x)$ using a $\epsilon u$-approximate quantile summary $Q$ of the data stream \big{\{}x^{i}_{(k)}\big{\}}_{i=1}^{n}. This is a simple extension to the proof in 17 for inversely querying a $\epsilon$-approximate quantile summary. Firstly, let $\hat{F}_{n,(k)}(x)=i/n$, meaning $\tilde{x}^{i}_{(k)}\leq x<\tilde{x}^{i+1}_{(k)}$, where $\tilde{x}^{0}_{(k)}=-\infty$ and $\tilde{x}^{n+1}_{(k)}=\infty$. Let $j\in[1,L]$, then we know that using the quantile summary we keep an approximation, $\tilde{x}^{i_{j}}_{(k)}$, to the $i$’th order statistic of the data stream; this approximation satisfies $\tilde{x}^{i_{j}}_{(k)}\leq\tilde{x}^{i}_{(k)}\leq x<\tilde{x}^{i+1}_{(k)}\leq\tilde{x}^{i_{j+1}}_{(k)}$. Note that as the summary is $\epsilon u$-approximate, we have $\lvert i_{j}-i\rvert\leq 2\epsilon\min(i,n-i)$. Also recall from Sec. 3.1 that we can only access the minimum and maximum values that $i_{j}$ can take, and not actually $i_{j}$ itself. To find an approximation to $i_{j}$ we can simply search all values $v_{l}$ in $Q$, for $l=1,\ldots,L$, for the $l$ that satisfies $v_{l}\leq x<v_{l+1}$ (with $v_{L+1}=\infty$) and take $(r_{min,Q}(v_{l})+r_{max,Q}(v_{l}))/2$ as $\hat{i}_{j}$ as the approximation to $i_{j}$. If $x<v_{1}$, of course take $\hat{i}_{j}=0$. Now we know that $r_{max,Q}(v_{l})-r_{min,Q}(v_{l})\leq r_{max,Q}(v_{l})-r_{min,Q}(v_{l-1})\leq 2\epsilon\min(i,n-i)$, and therefore $\lvert\hat{i}_{j}-i_{j}\rvert\leq\epsilon\min(i,n-i)$. Due to the triangle inequality we have that $\lvert\hat{i}_{j}-i\rvert\leq 3\epsilon\min(i,n-i)$ and finally that

[TABLE]

4.3 Bounding the error of the approximation to the tail dependence coefficients

Now that we have covered some necessary bounds, we can derive the guaranteed error bound of the modified copula summary and therefore the tail dependence coefficients approximations. Recall that the $\epsilon u$-approximate empirical copula approximation is given by,

[TABLE]

therefore the approximation to the lower tail dependence coefficient (for a fixed $i$) is given by,

[TABLE]

and the approximation to the upper tail dependence coefficient (for a fixed $j=n-i$) is given by,

[TABLE]

Theorem 4.1**.**

Let $\tilde{\tilde{C}}(i/n,i/n)$ denote the $\epsilon u$-approximate copula summary approximation, given in (25), to the bivariate empirical copula $\hat{C}(i/n,i/n)$, then one can bound the error of this approximation by,

[TABLE]

Therefore the approximation to the lower and upper tail dependence functions in (26) and (27) can be bounded by,

[TABLE]

and

[TABLE]

respectively. The approximation error is therefore constant with increasing $n$.

To prove this bound, we shall follow the steps of the proof in 15 with some modifications.

Proof 4.2**.**

We shall split the error

[TABLE]

into three contributing parts via the triangle inequality and prove each individually,

[TABLE]

The first contributing part in (31) corresponds to the error associated with using an inverse query of the $\epsilon u$-approximate quantile summary $M(S_{(2)}^{1},\ldots,S_{(2)}^{E})$, instead of the empirical CDF $\hat{F}_{\hat{n}_{1},(2)}$.

Theorem 4.3**.**

The term (A) in (31) can be bounded by,

[TABLE]

Proof 4.4**.**

From (24), we can bound

[TABLE]

Let

[TABLE]

First we will concentrate on bounding this term by above and below, since (32) is proportional to the minimum of $D$ and $1-D$. As $D$ depends on the query $\tilde{\tilde{F}}_{n,(2)}^{-1}(i/n)$, this query also needs to be bounded above and below.

Bounding $D$ by above

Note that the query $\tilde{\tilde{F}}_{n,(2)}^{-1}(i/n)$ is an increasing function with $i$, and clearly $\hat{F}_{\hat{n}_{1},(2)}(\tilde{\tilde{F}}_{n,(2)}^{-1}(i/n))$ is increasing with $\tilde{\tilde{F}}_{n,(2)}^{-1}(i/n)$ and $i$ too. Therefore it suffices to bound $D$ with $\hat{F}_{\hat{n}_{1},(2)}$ evaluated at the upper bound for $\tilde{\tilde{F}}_{n,(2)}^{-1}(i/n)$. Note that as $M(S_{(2)}^{1},\ldots,S_{(2)}^{L})$ is a $\epsilon u$-approximate summary as shown in Sec. 4.1, we have that,

[TABLE]

and therefore,

[TABLE]

Recall that $\hat{n}_{1}\hat{F}_{\hat{n}_{1},(2)}(G)$ is just the count of all elements in \big{\{}x_{(2)}^{\hat{I}(i)}\big{\}}_{i=1}^{\hat{n}_{1}} less than or equal to $\tilde{x}_{(2)}^{i+\lfloor\epsilon\min(i,n-i)\rfloor}$. Since \big{\{}x_{(2)}^{\hat{I}(i)}\big{\}}_{i=1}^{\hat{n}_{1}}\subset\big{\{}x_{(2)}^{i}\big{\}}_{i=1}^{n}, it must follow that $\hat{n}_{1}\hat{F}_{\hat{n}_{1},(2)}(G)\leq n\hat{F}_{n,(2)}(G)\leq i+\lfloor\epsilon\min(i,n-i)\rfloor$. Therefore,

[TABLE]

Bounding $D$ by below

For a lower bound on $D$, we shall use the following bound222This is given by the fact that in the case where there is a perfect negative rank correlation between the two components in the data stream \big{\{}x_{(1)}^{i},x_{(2)}^{i}\big{\}}_{i=1}^{n}, there will always be $\max(2i-n,0)$ values of $j$ that satisfy $x_{(1)}^{j}\leq\tilde{x}_{(1)}^{i}$ and $x_{(2)}^{j}\leq\tilde{x}_{(2)}^{i}$,

[TABLE]

Now, recall from Sec. 3.4 that $\hat{I}\subset I$ if $\hat{n}_{1}<n_{1}$ or $I\subset\hat{I}$ if $\hat{n}_{1}>n_{1}$; given that $S_{(1)}$ is a $\epsilon u$-approximate summary we have that,

[TABLE]

In addition to this, note that as $M(S_{(2)}^{1},\ldots,S_{(2)}^{L})$ is a $\epsilon u$-approximate summary, we have that $\tilde{\tilde{F}}_{n,(2)}^{-1}(i/n)\geq\tilde{x}_{(2)}^{i-\lceil\epsilon\min(i,n-i)\rceil}$ and,

[TABLE]

Then finally,

[TABLE]

and therefore,

[TABLE]

Now that $D$ and $1-D$ have been bounded from above, these bounds can be used to bound the overall error in (32).

Overall bound

The term (A) in (31) can be bounded by,

[TABLE]

The second part of the error in (31) corresponds to the error associated with evaluating the empirical CDF $\hat{F}_{\hat{n}_{1},(2)}$ with the query of the $\epsilon u$-approximate quantile summary $M(S_{(2)}^{1},\ldots,S_{(2)}^{L})$, instead of the empirical quantile function $\hat{F}^{-1}_{n,(2)}$.

Theorem 4.5**.**

The term (B) in (31) can be bounded by,

[TABLE]

Proof 4.6**.**

This can be proved in the same way as Theorem 3 was in 15, only with a $\pm\epsilon\min(i,n-i)$ error from the $\epsilon u$-approximate summaries rather than the $\pm\epsilon n$ error from the $\epsilon$-approximate summaries.

Finally, the third part of the error in (31) corresponds to the error associated with using the cardinality of the set $\hat{I}$ in Sec. 3.4, $\hat{n}_{1}$, instead of the cardinality of the set $I$ in Sec. 2, $n_{1}$.

Theorem 4.7**.**

The term (C) in (31) can be bounded by,

[TABLE]

Proof 4.8**.**

This can be proved in the same way as Theorem 4 was in 15, only with a $\pm\epsilon\min(i,n-i)$ error from the $\epsilon u$-approximate summaries rather than the $\pm\epsilon n$ error from the $\epsilon$-approximate summaries.

4.4 Space-memory

A single $\epsilon u$-approximate quantile summary is of the worst case length L=\mathcal{O}\big{(}\frac{\log^{2}(\epsilon n)}{\epsilon}\big{)}, since there is no benefit in using $u<1/(\epsilon n)$ 7. Therefore the worst case length of the modified copula summary is L^{2}=\mathcal{O}\big{(}\frac{\log^{4}(\epsilon n)}{\epsilon^{2}}\big{)}. This bound is not saturated in all cases; for instance it was shown in 13 that for some random streams the space-efficient summaries used in this paper can have size independent of $n$. The saturation of the worst case length of the $\epsilon u$-approximate summaries that make up the modified copula summary, given above, is discussed in 27. For streams that are increasing in value for example, the space memory used by the $\epsilon u$-approximate quantile summaries is increasing with the length of the data stream. For the scope of this paper, this issue is not investigated further. However there are alternative approaches to relative error approximations of empirical quantile functions proposed in the literature; see 27 for more details on these.

It may be of interest to the reader to adapt this methodology to studying the tail dependence of higher dimensional data streams. This would be achieved by approximating higher dimensional empirical copulas, and then substituting these approximations into the tail dependence coefficient expressions in (10) and (11). Unfortunately a natural extension of the copula summary structure presented here to higher dimensions would incur an exponentially growing (with dimension) space-memory requirement 16. Turn to the work in 15 for an example of how one may utilise bivariate copula summaries to estimate higher dimensional empirical copulas via decompositions involving a sequence of bivariate empirical copulas 1, and alleviate this issue.

5 Simulations and case-study

This section will provide some numerical demonstrations of the tail dependence coefficient estimation scheme in the streaming data regime, proposed in this paper. The theoretical analysis given throughout the paper is also numerically supported here. In Sec. 5.1, the estimation of the tail dependence coefficients for random variables observed through data streams will be considered. Then in Sec. 5.2, the properties of the modified space-efficient copula summary (utilised for the estimation of the tail dependence coefficients) will be numerically investigated. Finally, a case-study of the Los Alamos National Laboratory (LANL) netflow data-set is considered.

5.1 Approximation to the tail dependence coefficients

First, a demonstration of estimating both of the tail dependence coefficients between two random variables observed in a bivariate stream of data, will be presented. Consider the data stream \big{\{}x_{(1)}^{i},x_{(2)}^{i}\big{\}}_{i=1}^{n}, where $n=3\times 10^{5}$. Both components are randomly sampled from $\mathcal{N}(0,1)$ in one case, or $\text{Beta}(10,1)$ in another. In both cases the components are correlated with Pearson’s correlation $\rho=0.8$. The accuracy parameter $\epsilon$ takes the value of $0.1$, and the copula summary presented in this paper is constructed for five independent bivariate data streams for each distribution. We set $i=25$; this value is used to compute the estimate to both of the tail dependence coefficients using the copula summaries. These estimates are computed after every 5000’th element has been added to the data streams. The absolute error, away from the empirical lower and upper tail dependence coefficients of all data streams sampled from the Gaussian distribution, are shown over time in Figures 2 and 2 respectively for the modified copula summary proposed in this paper. The error of the approximations to these tail dependence coefficients for all data streams sampled from the Beta distribution are also shown in Figures 4 and 4. Visible in all plots is the theoretical stream-length invariant bound presented in (29) and (30). The error for the upper tail dependence coefficient is slightly more than that of the lower tail dependence coefficient in the case of the data streams sampled from the Gaussian distribution, and vice-versa for the data streams sampled from the Beta distribution.

5.2 Properties of the copula summary

Next the properties of the modified copula summary, including it’s space-efficiency and implementation runtime, will be numerically demonstrated. Consider the modified copula summary presented in this paper, with $\epsilon=0.1$, used in the previous numerical experiment (where five independent bivariate data streams are sampled from both a Gaussian distribution and a Beta distribution). The absolute error of the approximation from this copula summary over time and all data streams sampled from the Gaussian distribution, for the evaluation points $(u_{1},u_{2})=(0.7,0.7)$ and $(u_{1},u_{2})=(0.02,0.02)$, are shown in Figures 6 and 6 respectively. The same error, only for the data streams sampled from the Beta distribution, are shown in Figures 8 and 8. These show that, as with the standard copula summary presented in 15, the error of the empirical copula approximation is bounded by a constant. However now, in the case where the evaluation point $(0.7,0.7)$ is not in the tails of each marginal, the approximation has a higher bound (and therefore exhibits greater numerical error) than in the case where the evaluation point $(0.02,0.02)$ is in the tails. This is in line with the theoretical analysis presented in (28). Next, the implementation runtime and space-efficiency of the modified copula summary, over all data streams sampled from the Gaussian distribution, is demonstrated. Figure 10 shows the runtime (in seconds) of an iteration of the copula summary algorithm after every 5000’th element is added to the streams. Occassional peaks are due to the combination operation in Sec. 3.3 being implemented. Similarly, Figure 10 shows the size ratio of the modified copula summary to the entire streams:

[TABLE]

after every 5000’th element is added to the streams. This shows the increasing space-efficiency of the copula summary proposed in this paper, used for approximations to the empirical tail dependence coefficients, as the stream length increases. At the end of the data stream the size of the entire stream is 5440520 bytes, 39 times the size of the copula summary.

Finally, the effect of $\epsilon$ on the approximation error of the modified copula summary presented in this paper is explored. The absolute error of the approximation from the copula summary at the evaluation points $(u_{1},u_{2})=(0.7,0.7)$ and $(u_{1},u_{2})=(0.02,0.02)$, over five independent data streams of length 30000 (sampled from the aforementioned Gaussian distribution), is shown for a number of different $\epsilon$ values in Figure 11. The error was computed after every element has been added to the data streams. As found for a fixed value of $\epsilon$ in Figures 6 and 6, the copula approximation error is lower for the evaluation point in the tail, $(0.02,0.02)$. In addition to this, as $\epsilon$ decreases, the approximation error at both evaluation points decreases respectively too; this behaviour is described in the theoretical bound presented in (28).

5.3 LANL netflow data

This section applies the proposed methodology to a case-study of cyber-netflow traffic data consisting of records of computer network communication, available freely from Los Alamos National Laboratory (LANL, https://csr.lanl.gov/data/2017.html). This provides an example of streaming data with great importance in the field of cyber-security 2. Various aspects of this data are throughly described in 26. It has been shown that flow based techniques have a number of computational advantages and are successful in detecting a variety of malicious network behaviors 25. Daily netflow data is available for 89 days (indexed as Day 2 to Day 91, starting with Day 2). Each flow records an aggregate summary of a bi-directional network communication between any two of approximately 60000 devices of the LANL network 26. The aggregate summary consists of various records on the communication for every second of the day (given in epoch time format). We consider the variable SrcPackets, the number of bytes the SrcDevice sent during a certain communication event, where SrcDevice is the device that likely initiated the event.

Modeling the records of communication events in a fixed time window is a common approach to understand normal behaviour of a cyber network, and in turn helps in detecting unusual behaviour 9. This type of behaviour could be detected by studying outlying extreme values of either variable mentioned above, however these particular values might be highly correlated with extrema from the other variable. By knowing the tail dependence of the variables through the tail dependence coefficients, one could reason about how likely extrema occuring for both variables at the same time signifies unusual behaviour. In this application, we consider the above data in the streaming sense over a fixed time window of one second (lowest time resolution available in the netflow data-set). Then for a given day, we consider the bivariate data stream \big{\{}x_{(1)}^{i},x_{(2)}^{i}\big{\}}, where $x_{(1)}^{i}$ is the total volume of bytes transferred during events that initiated at time $i$ and $x_{(2)}^{i}$ is the total number of active edges in the network at time $i$, $i=1,\ldots,86400$. In the following analysis of the netflow data, we use log transform values of both the variables. Figure 12 shows scatter plots of \big{\{}x_{(1)}^{i},x_{(2)}^{i}\big{\}}, $i=1,\ldots,86400$, for two randomly selected days, namely Day 3 and Day 10. The points in the red highlight those lying below the 0.005th sample quantile of both marginals, whereas the darkened points highlight those lying in the 0.005th sample quantile of either marginal. In both of the days, the pattern indicates a positive co-movement in the lower tails.

Figure 13 shows the empirical lower tail dependence coefficient between the total volume of bytes transferred and the number of active edges over Day 10, for different values of $i$. As one can see, there is significant tail dependence demonstrated by a non-zero coefficient as $i$ tends to 0. Also shown in this Figure is the approximation to the lower tail dependence coefficient computed using the proposed copula summary in this paper, with $\epsilon=0.075$. The copula summary is constructed sequentially after each element is added to the data stream on each second of Day 10, in the manner described in Sec. 3.1. This represents the summary that would be maintained if this data was to be streamed in real-time to a netflow traffic analyzer. The approximation shown in Figure 13 is computed after the entire data stream from Day 10 has been inserted into the copula summary; it shows the significant non-zero tail dependence coefficient well for all $i$ values considered. The average time for each element to be inserted into the copula summary was just 0.03 seconds; therefore this algorithm would be sufficiently fast to match the data acquisition rate (one second). The total number of elements stored in the copula summary was 26448, which is much less than the number of elements in the data stream (172800), and results in a size ratio of 0.15. Therefore this approximation of the lower tail dependence coefficient is space-efficient relative to storing the entire data stream.

To see the impact of the condition in (16) on the structure of the copula summary applied to approximating the lower tail dependence coefficient for this netflow traffic data, Figure 15 shows the values of $g_{(1)}^{i}$, for $i=1,\ldots,L$, defined in (15), after all elements have been added to the data stream. Recall these values are the $g$ values within the tuples making up the $\epsilon u$-approximate quantile summary $S_{(1)}$ in the copula summary. In addition to this, Figure 15 shows the values of $L_{i}$ (the subsummary $S_{(2)}^{i}$ lengths), for $i=1,\ldots,L$. These show a similar pattern to the values of $g_{(1)}^{i}$; where there is only very few elements being represented by the $i$’th tuple in $S_{(1)}$, there is also only very few elements to have entered or been merged into the corresponding subsummary $S_{(2)}^{i}$. This illustrated behaviour of the copula summary, that makes up the approximations to the lower tail dependence coefficient of the netflow traffic data, demonstrates the benefits of the modifications presented in this paper.

6 Summary and conclusion

This article has presented a stream-length invariant bounded and space-efficient approximation to empirical tail dependence coefficients for bivariate streaming data. This regime of data means that an indefinite set of data can’t be stored in its entirety, and analyzes of the data (such as modelling the tail dependence) need to be updated on-the-fly. The approximation presented in this paper is implemented via the use of a modified copula summary; the standard copula summary was introduced recently in 15. The modification, first introduced in the context of quantile estimation 7, allows the error of the approximation to be refined in the tails of the copula marginals, and therefore it does not grow linearly with the number of elements in the data stream (such as the case is when using the standard copula summary).

The methodology presented in this paper is for use with bivariate streaming data. An example of how the copula summaries, that make up the approximation to empirical tail dependence coefficients in this paper, could be extended to higher dimensional data streams was presented in 15. Due to the wide-range of industries that now use streaming data, developing the techniques surrounding dependence modelling for this type of data is important; this paper continues this line of work. A relevant case-study of such an industry (cyber-security) is considered at the end of this paper; the proposed methodology is employed to space-efficiently capture lower tail dependence in a netflow data-set from the Los Alamos National Laboratory.

Appendix A Proof of bound on error for $\epsilon u$-approximate quantile summary approximations

This section shows that the condition in (16),

[TABLE]

guarantees that a $\epsilon u$-approximate summary $Q$ can return an approximation $\tilde{x}_{(k)}^{j}$, to $\hat{F}_{n,(k)}(u)=\tilde{x}_{(k)}^{\lceil un\rceil}$ for $k=1,2$, where $j\in[\lceil un\rceil-\epsilon\min(u,(1-u))n,\lceil un\rceil+\epsilon\min(u,(1-u))n]$. This is a generalisation of the analysis in 7. First, we must show that the insertion and combining operations in Sec. 3.2 and 3.3 do not alter the bound in (16). Start by considering the insertion operation outlined in Sec. 3.2. When this operation is carried out and the added tuple gets input after $v_{i+1}$, the first term in the minimum is unaffected. The second term in the minimum in this case is also unaffected, since $r_{max,Q}(v_{i+1})$ increases by 1 ($g=1$) but so does $n$; this means that the bound is still satisfied. If the added tuple gets input before $v_{i}$, then the first term in the minimum increases by 1; this also means that the bound is still satisfied. The second term in the minimum in this case is again unaffected since both $r_{max,Q}(v_{i+1})$ and $n$ increases by 1. Therefore the insertion operation does not affect the bound in (16). Next consider the combining operation in Sec. 3.3. Clearly this operation only combines tuples in $Q$ when the condition in (16) is satisfied and therefore this operation does not alter the required bound.

Finally, we now show that $Q$ can be queried to return a value $j$ where $j\in[\lceil un\rceil-\epsilon\min(u,(1-u))n,\lceil un\rceil+\epsilon\min(u,(1-u))n]$. Assume for simplicity that $\lceil un\rceil=un$, however the derivation below can simply be adjusted to relax this assumption. Let $r_{i}=r_{max,Q}(v_{i+1})-r_{min,Q}(v_{i})$ and also let $i$ be the smallest index that satisfies

[TABLE]

Therefore, we have $r_{i-1}+g^{i-1}+\Delta^{i-1}\leq\min\left((1+\epsilon)un,\epsilon n+(1-\epsilon)un\right)$. Now, note that given the form of $r_{i}$ we have,

[TABLE]

So now combining this with (33) we have that $(1+2\epsilon)r_{i}>\min\left((1+\epsilon)un,\epsilon n+(1-\epsilon)un\right)$, and

[TABLE]

So overall we have,

[TABLE]

By the construction of $Q$ we know that the value $l$ that satisfies $\tilde{x}_{(k)}^{l}=v_{i}$, lies in between $r_{i}+g^{i}$ and $r_{i}+g^{i}+\Delta^{i}$. Therefore we need to show that (b) $-$ (a) $\leq 2\epsilon\min\left(un,(1-u)n\right)$. If $u\leq 0.5$, then $(1-\epsilon)un$ is the minimum of (a) and $(1+\epsilon)un$ is the minimum of (b), and therefore,

[TABLE]

If $u>0.5$, then $(1+\epsilon)un-\epsilon n$ is the minimum of (a) and $(1-\epsilon)un+\epsilon n$ is the minimum of (b), and therefore

[TABLE]

∎