Weak Convergence of Stationary Empirical Processes

TL;DR

This paper extends the weak convergence results of empirical processes to more general, function-indexed processes under broad dependence conditions, providing a unified framework for stationary empirical processes.

Contribution

It introduces a general weak convergence result for empirical processes indexed by functions of bounded variation, applicable under alpha mixing dependence.

Findings

Weak convergence established for processes indexed by functions of bounded variation.

Applicable to stationary sequences with alpha mixing dependence.

Extends classical empirical process results to more general settings.

Abstract

We offer an umbrella type result which extends weak convergence of the classical empirical process on the line to that of more general processes indexed by functions of bounded variation. This extension is not contingent on the type of dependence of the underlying sequence of random variables. As a consequence we establish weak convergence for stationary empirical processes indexed by general classes of functions under alpha mixing conditions.

Equations

$${\bar{\mathbb{Z}}}_n(g)=\frac{1}{\sqrt{n}}\sum _{i=1}^n\left\{g(X_i)-\mathbb{E}g(X_i)\right\},g\in \mathcal{G},$$

$${\mathbb{G}}_n(t)=\frac{1}{\sqrt{n}}\sum _{i=1}^n\left\{1_{(-\infty ,t]}(X_i)-F(t)\right\},$$

$$||g|{|}_{TV}=\underset{\Pi }{\sup }\sum _{x_i,x_{i+1}\in \Pi }|g(x_{i+1})-g(x_i)|.$$

$$B{V}_T:=\{g:\mathbb{R}\to \mathbb{R}:\Vert g{\Vert }_{TV}\le T,\Vert g{\Vert }_{\infty }\le T\}.$$

$$\left\{{\overline{\mathbb{Z}}}_n(g):=\int g(x)\mathrm{d}{\mathbb{Z}}_n(x),g\in \mathcal{G}\right\}$$

$$\underset{g\in B{V}_T}{\sup }\underset{h\in B{V}_T^{\prime }}{\inf }\left|{\bar{\mathbb{Z}}}_n(g)-{\bar{\mathbb{Z}}}_n(h)\right|\le T\underset{x}{\sup }|{\mathbb{G}}_n(x)-{\mathbb{G}}_n(x^{-})|.$$

$$\left|\int g\mathrm{d}{\mathbb{G}}_n-\int \overline{g}\mathrm{d}{\mathbb{G}}_n\right|$$

$$\mathbb{E}[\bar{\mathbb{Z}}(f)\bar{\mathbb{Z}}(g)]$$

$${\mathbb{G}}_n^{\ast }(t)=\sqrt{m_n}\left(\frac{1}{m_n}\sum _{i=1}^{m_n}{1}_{(-\infty ,t]}(X_i^{\ast })-\frac{1}{n}\sum _{i=1}^n{1}_{(-\infty ,t]}(X_i)\right),t\in \mathbb{R},$$

$$d_{BL}({\mathbb{Z}}_n^{\ast },\mathbb{Z})=\underset{h\in B{L}_1}{\sup }\left|{\mathbb{E}}^{\ast }[h({\mathbb{Z}}_n^{\ast })]-\mathbb{E}[h(\mathbb{Z})]\right|$$

$${\tilde{\mathbb{Z}}}_n(g)=-\int {\mathbb{Z}}_n\mathrm{d}g$$

$${\bar{\mathbb{Z}}}_n(g)={\tilde{\mathbb{Z}}}_n(g)+R_n(g)$$

$$R_n(g)\le T\underset{x}{\sup }|{\mathbb{Z}}_n(x)-{\mathbb{Z}}_n(x^{-})|\to 0,$$

$$d_{BL}({\bar{\mathbb{Z}}}_n,\tilde{\mathbb{Z}})$$

$$\int g(x)\mathrm{d}{\mathbb{Z}}_n(x)=T_1({\mathbb{Z}}_n)+T_2({\mathbb{Z}}_n)$$

$$T_1({\mathbb{Z}}_n)$$

$$T_2({\mathbb{Z}}_n)$$

$$T_2({\mathbb{Z}}_n)=\sum _i{\alpha }_i({\mathbb{Z}}_n(a_i)-{\mathbb{Z}}_n(a_i^{-}))$$

$${\Psi }_{\beta ,F_0}({\mathbb{Z}}_n):=$$

$$g_{\beta ,F_0}({\mathbb{Z}}_n):=$$

$$|{\bar{\mathbb{Z}}}_n(g)|\le \left\{2{\beta }^{-1}||g|{|}_{L_1(F_0)}+6||g|{|}_{TV}\right\}{\Psi }_{2\beta ,F_0}({\mathbb{Z}}_n)+{\beta }^{-1}\Vert g{\Vert }_{L_1(F_0)}{g}_{\beta ,F_0}({\mathbb{Z}}_n).$$

$$F_0(s_j)-F_0(s_{j-1})\ge \beta ,F_0(s_M)<1-2\beta$$

$$F_0(s_j^{-})-F_0(s_{j-1})\le 2\beta ,$$

$${\tilde{\mathbb{Z}}}_n(t):=\sum _{i=1}^M{\mathbb{Z}}_n(s_{j-1})1_{[s_{j-i,}{s}_j)}(t)$$

$$\underset{x}{\sup }|{\mathbb{Z}}_n(x)-{\tilde{\mathbb{Z}}}_n(x)|$$

$$\int g(s)\mathrm{d}{\mathbb{Z}}_n(s)=\int g(s)\mathrm{d}{\tilde{\mathbb{Z}}}_n(s)+\int g(s)\mathrm{d}({\mathbb{Z}}_n-{\tilde{\mathbb{Z}}}_n)(s).$$

$$\int g(s)\mathrm{d}({\mathbb{Z}}_n-{\tilde{\mathbb{Z}}}_n)(s)=-\int ({\mathbb{Z}}_n-{\tilde{\mathbb{Z}}}_n)(s)\mathrm{d}g(s)+\int \int 1_{x=y}\mathrm{d}g(x)\mathrm{d}({\mathbb{Z}}_n-{\tilde{\mathbb{Z}}}_n)(y).$$

$$\left|\int \int 1_{x=y}\mathrm{d}g(x)\mathrm{d}({\mathbb{Z}}_n-{\tilde{\mathbb{Z}}}_n)(y)\right|$$

$$\left|\int g(s)\mathrm{d}({\mathbb{Z}}_n-{\tilde{\mathbb{Z}}}_n)(s)\right|\le 3{\Vert g\Vert }_{TV}{\Psi }_{2\beta ,F_0}({\mathbb{Z}}_n)$$

$$\int g(s)d{\tilde{\mathbb{Z}}}_n(s)=\sum _{j=1}^{M}g(s_j)({\mathbb{Z}}_n(s_j)-{\mathbb{Z}}_n(s_{j-1})).$$

$$\left|\int g(s)d{\tilde{\mathbb{Z}}}_n(s)\right|\le \left|\sum _{j=1}^{M}g(s_j)({\mathbb{Z}}_n(s_j^{-})-{\mathbb{Z}}_n(s_{j-1}))\right|+\left|\sum _{j=1}^{M}g(s_j)({\mathbb{Z}}_n(s_j)-{\mathbb{Z}}_n(s_j^{-}))\right|$$

Full text

Running title: Weak convergence of stationary empirical processes

MSC2000 Subject classification: Primary 60F17; secondary 60G99.

Keywords and phrases: Integration by parts, bounded variation, empirical processes, stationary distributions, weak convergence.

1

Introduction

We consider the empirical process

[TABLE]

indexed by some class $\mathcal{G}$ of functions $g:\mathbb{R}\rightarrow\mathbb{R}$. It is an obvious generalization of the classical process

[TABLE]

in which case the indexing class is $\mathcal{G}=\left\{g(x)=1_{(-\infty,t]}(x):\ t\in\mathbb{R}\right\}$. If the underlying sequence $\{X_{i}\}_{i\geq 1}$ is i.i.d., then the limiting behavior (as $n\rightarrow\infty$) of the empirical processes $\{\bar{\mathbb{Z}}_{n}(g),\ g\in\mathcal{G}\}$ is well understood. The same is true for the bootstrap counterpart $\{\bar{\mathbb{Z}}_{n}^{\ast}(g),\ g\in\mathcal{G}\}$ based on an i.i.d. bootstrap sample $\{X_{1,n}^{\ast},\ldots,X_{m_{n},n}^{\ast}\}$, see, for instance, Van der Vaart and Wellner (1996) and Dudley (1999). The theory of weak convergence of empirical processes based on independent sequences has yielded a wealth of statistical applications and, in particular, it was instrumental for establishing the weak convergence of numerous novel statistics. Often the limiting distributions of these statistics do not allow for closed form solutions, in which case the bootstrap version of the process is utilized. As is standard practice nowadays, see the thorough exposition in the first chapter of Van der Vaart and Wellner (1996), weak convergence in this paper should be understood in the sense of Hoffmann-Jørgensen and expectations should be interpreted as outer expectations.

For empirical processes based on stationary sequences $\{X_{i}\}_{i\geq 1}$ the situation is rather different. The classical process $\{\mathbb{G}_{n}(t),\,t\in\mathbb{R}\}$ has been treated by numerous authors, who established weak convergence under sharp mixing assumptions, see, for instance, Rio (2000). However, nothing similar exists for more general processes. The only work that we could find in the literature, Andrews and Pollard (1994), treats more general indexing classes, but imposes rather restrictive assumptions on the decay of $\alpha$-mixing coefficients. This discrepancy between the conditions needed for $\bar{\mathbb{Z}}_{n}(g)$ and those for $\mathbb{G}_{n}(t)$, is due to fact that the typical approach for proving the uniform limiting theorems heavily relies on the estimation of entropy numbers, which in turn require good exponential maximal inequalities. Only $\beta$-mixing, via decoupling, allows for such an estimate. This is the reason why one can find in the literature the treatment of $\bar{\mathbb{Z}}_{n}(g)$ only for $\beta$-mixing sequences, see Arcones and Yu (1994) and Doukhan, Massart and Rio (1995).

The situation for bootstrap of stationary empirical processes is even worse. Although introduced more than twenty years ago, we only found three papers that study the bootstrap for stationary empirical processes indexed by general classes $\mathcal{G}$, see Künsch (1989), Liu and Singh (1992) and Sengupta, Volgushev and Shao (2016). All these results operate under $\beta$-mixing assumptions. Moreover, only in the case of VC-classes do we have the sharp conditions, see Radulović (1996). Bracketing classes were considered by Bühlmann (1995), but this was established under restrictive assumptions on both $\beta$-mixing coefficients and bracketing numbers. It is worth mentioning that the covariance function of the limiting Gaussian process is unknown, and consequently in most actual applications of these results, we heavily rely on the bootstrap version of the process for which adequate results are sorely lacking. In short, the most general $\alpha$-mixing sequences have never been considered for stationary bootstrap processes $\{\bar{\mathbb{Z}}_{n}^{\ast}(g),\ g\in\mathcal{G}\}$, while for the non-bootstrap version $\{\bar{\mathbb{Z}}_{n}(g),\ g\in\mathcal{G}\}$ we have only one example in the literature.

In what follows we prove two general results that allow us to extend weak convergence of $\{\mathbb{G}_{n}(t),\ t\in\mathbb{R}\}$ to that of $\{\bar{\mathbb{Z}}_{n}(g),{g\in\mathcal{G}}\}$, where $\mathcal{G}$ is a class of functions of uniformly bounded total variation (called $BV_{T}$ in this paper). We would like to point out that although there are examples of Donsker classes of infinite variation (for instance, $f:[0,1]\rightarrow[0,1]$ with $|f(x)-f(y)|\leq|x-y|^{\alpha}$, $1/2<\alpha<1)$, such cases are rather the exception than the norm. The majority of examples of bounded Donsker classes that are given in the literature are subsets of the class $BV_{T}$.

This enlargement from $\{\mathbb{G}_{n}(t),{t\in\mathbb{R}}\}$ to $\{\bar{\mathbb{Z}}_{n}(g),\ {g\in\mathcal{G}}\}$ is not contingent on the dependence structure of underlying sequence $\{X_{i}\}_{i\geq 1}$ (or $\{X_{i}^{\ast}\}_{i\geq 1}$) and the only requirement is that $\mathbb{G}_{n}(t)$ converges weakly to a Gaussian process. This allows us to derive weak convergence of $\{\bar{\mathbb{Z}}_{n}(g),\ {g\in\mathcal{G}}\}$ and $\{\bar{\mathbb{Z}}_{n}^{\ast}(g),\ {g\in\mathcal{G}}\}$ for $\alpha$-mixing sequences. The same extension applies for the short memory causal processes considered in Doukhan and Surgailis (1998), as well as processes treated in Dehling, Durieu and Volny (2009). The technique we employ is a simple application of the integration-by-parts formula, followed by the continuous mapping theorem. Arguably this approach may have been known before, although we could not find any communication of it in the literature, and certainly not among the research related to stationary empirical processes. A follow-up paper (Radulović, Wegkamp and Zhao (2017)) extends the results obtained in this work to classes indexed by multivariate functions of bounded variation, with an emphasis on empirical copula processes. An important technical difference with the follow-up paper is that here we allow for general stationary distribution functions of the underlying $\{X_{i}\}_{i\geq 1}$. The proof for general, non-continuous processes, is not trivial caused by technical complications related to the interplay between the atoms of the limiting process $\mathbb{Z}(t)$ and discontinuities of $g\in\mathcal{G}$.

The paper is organized as follows. Section 2 contains the statement of the main result (Theorem 1) and related discussions, while Section 3 presents the bootstrap version (Theorem 6) of this result. The proofs of these two main results are collected in Section 4. For completeness, the well-known integration by parts formula can be found in the appendix.

2 Main results

For the total variation norm of a function $g:\mathbb{R}\rightarrow\mathbb{R}$, we use the notation

[TABLE]

The supremum is taken over all countable partitions $\Pi=\{x_{1}<x_{2}<\ldots\}$ of $\mathbb{R}$. We set, for $T>0$,

[TABLE]

We let $BV_{T}^{\prime}\subset BV_{T}$ be the class of all functions $g$ in $BV_{T}$ that are right-continuous. Finally, we let $\{\mathbb{Z}_{n}(t),\ t\in\mathbb{R}\}$ be an arbitrary stochastic process such that

• A1:

$\lim_{|t|\rightarrow\infty}\mathbb{Z}_{n}(t)=0$;

• A2:

The sample paths of $\mathbb{Z}_{n}$ are right-continuous and of bounded variation.

Clearly, both requirements A1 and A2 are met for the canonical empirical process $\{\mathbb{G}_{n}(t),\ t\in\mathbb{R}\}$. In this paper, we study the limit distribution (as $n\rightarrow\infty$) of the sequence of processes

[TABLE]

for some class $\mathcal{G}\subseteq BV_{T}^{\prime}$, for some finite $T$.

Although the motivation as well as the most notable applications of our results are related to the canonical case $\mathbb{Z}_{n}(x)=\mathbb{G}_{n}(x)$, the actual proof carries over for any process $\mathbb{Z}_{n}$ as long as assumptions A1 and A2 are satisfied.

Our main result is the following theorem.

Theorem 1

*Assume that $\{\mathbb{Z}_{n}(t),\ t\in\mathbb{R}\}$ converges weakly, as $n\rightarrow\infty$, to a Gaussian process $\{\mathbb{Z}(t),\ t\in\mathbb{R}\}$, that has uniformly continuous sample paths with respect to the distance $d(s,t)=|F_{0}(s)-F_{0}(t)|$ for some distribution function $F_{0}$.

Then, for any $T<\infty$ and $\mathcal{G}\subseteq BV_{T}^{\prime}$, $\{\bar{\mathbb{Z}}_{n}(g),\ g\in\mathcal{G}\}$ converges weakly, as $n\rightarrow\infty$, to a Gaussian process in $\ell^{\infty}(\mathcal{G})$, that has uniformly $L_{1}(F_{0})$-continuous sample paths.*

Remark.

Usually, an envelope condition $\sup|g(x)|\leq G(x)$ and $G\in L_{2}(F_{0})$ is imposed. Since such a condition, coupled with $\|g\|_{TV}\leq T$, implies that the functions $g$ are uniformly bounded, we assume $\|g\|_{\infty}\leq T$ in our definition of $BV_{T}$.

Theorem 1 allows us to derive weak convergence of $\bar{\mathbb{Z}}_{n}(g)$ via $\mathbb{Z}_{n}(t)$, regardless of the structure of the latter process. For instance, taking $\mathbb{Z}_{n}$ as the standard empirical processes $\mathbb{G}_{n}$ based on stationary sequences $\{X_{i}\}_{i\geq 1}$, we obtain the following corollary as an immediate consequence of Theorem 1.

Corollary 2

Let $\{X_{k}\}_{k\geq 1}$ be a stationary sequence of random variables with distribution $F$ and $\alpha$-mixing coefficients $\alpha_{n}$ satisfying $\alpha_{n}=O(n^{-r})$, for some $r>1$ and all $n\geq 1$. Then, for any $\mathcal{G}\subseteq BV_{T}^{\prime}$,

$\{\int g\,{\rm d}\mathbb{G}_{n}$, $g\in\mathcal{G}\}$ converges weakly to a Gaussian process with uniformly $L_{1}(F)$-continuous sample paths in $\ell^{\infty}(\mathcal{G})$.

• -

For continuous $F$, $\mathcal{G}$ can be enlarged to $BV_{T}$.

Proof. It is well known, see Theorem 7.2, page 96 in Rio (2000), that $\alpha_{n}=O(n^{-r})$ with $r>1$, implies that the standard empirical process $\mathbb{G}_{n}(t)$ converges weakly to a Brownian bridge process with uniformly continuous paths with respect to the distance $d(s,t)=|F(s)-F(t)|$, for the stationary distribution $F$ of $X_{k}$. The result for \mathcal{G}\subseteq BV_{T}^{\prime}\,\now follows trivially from Theorem 1. If $F$ is continuous, then the limiting Brownian bridge has uniformly continuous sample paths with respect to the Lebesgue measure on $\mathbb{R}$. This, combined with Lemma 3 below, implies Corollary 2.

We used the following lemma.

Lemma 3

We have, for $\bar{\mathbb{Z}}_{n}(g)$ based on the canonical process $\mathbb{Z}_{n}(x)=\mathbb{G}_{n}(x)$,

[TABLE]

Proof. Let $g$ be an arbitrary function in $BV_{T}.$ We denote its countable many discontinuities by $a_{i}.$ Let $\overline{g}$ be the right-continuous version of $g$, that is, $\overline{g}(x)=g(x)$ for all $x\neq a_{i}$ and $\overline{g}(a_{i})=g(a_{i}^{+})$ for all $i$. Then

[TABLE]

and the conclusion follows.

We would like to point out that $\alpha$-mixing is the least restrictive form of available mixing assumptions in the literature. To the best of our knowledge, there are actually very few results that treat processes $\{\int g\,\mathrm{d}\mathbb{G}_{n},\,g\in\mathcal{G}\}$ indexed by functions, and they all require very stringent conditions on the entropy numbers of $\mathcal{G}$ and on the rate of decay for $\alpha_{k}$. See, for instance, Andrews and Pollard (1994). This is due to fact that $\alpha$-mixing does not allow for sharp exponential inequalities for partial sums. Consequently, the only known cases for which we have sharp conditions are under more restrictive, $\beta$-mixing dependence. Indeed, $\beta$-mixing allows for decoupling and it does yield exponential inequalities not unlike the i.i.d. case. The current state-of-the art results, Arcones and Yu (1994), Doukhan, Massart and Rio (1995), applied to bounded sequences, require $\sum_{n}\beta_{n}<\infty$.

While it is correct that $\alpha$-mixing is the weakest of the known mixing concepts, a referee pointed out that functionals of mixing processes constitutes a much weaker concept of short range dependence. For instance, Billingsley (1968, Theorem 22.2) establishes the empirical process CLT for certain functionals of $\phi$-mixing processes. The same referee made us aware that Theorem 1 also applies to the empirical process of long-range dependent data, established by Dehling and Taqqu (1989) for subordinate Gaussian processes, and by Ho and Hsing (1996) for linear processes. We provide an example to demonstrate that Theorem 1 goes beyond dependence defined via mixing conditions. It applies to short memory causal linear sequences $\{X_{i}\}_{i\geq 1}$ that are defined by $X_{i}=\sum_{j=0}^{\infty}a_{j}\xi_{i-j}$ based on i.i.d. random variables $\xi_{i}$ and constants $a_{i}$. While the $X_{i}$ form a stationary sequence, they do not necessarily satisfy any mixing condition. Weak convergence of the empirical processes $\{\mathbb{G}_{n}(t),\ t\in\mathbb{R}\}$ was established under sharp conditions in Doukhan and Surgailis (1998) on the weights $a_{j}$ ($\sum_{j\geq 0}|a_{j}|^{\gamma}<\infty$ for some $0<\gamma\leq 1$), characteristic function of $\xi_{0}$ ($|\mathbb{E}[\exp(iu\xi_{0})]|\leq C(1+|u|)^{-p}$ for all $u\in\mathbb{R}$, some $C<\infty$ and $2/3<p\leq 1$), and a moment condition on $\xi_{0}$ ($\mathbb{E}[|\xi_{0}|^{4\gamma}]<\infty$). To the best of our knowledge, there are no extensions to the more general processes $\{\bar{\mathbb{Z}}_{n}(g),\,g\in\mathcal{G}\}$. Theorem 1 and the Doukhan and Surgailis (1998) result combined imply the following:

Corollary 4

Let $\{X_{i}=\sum_{j\geq 0}a_{j}\xi_{i-j}\}_{i\geq 1}$ be such that conditions of Doukhan and Surgailis (1998, pp 87–88) are satisfied and let $F$ be the stationary distribution of $X_{i}$. Then, for any $\mathcal{G}\subseteq BV_{T}$, $\{\int g\,{\rm d}\mathbb{G}_{n},\ g\in\mathcal{G}\}$ converges weakly to a Gaussian process with uniformly $L_{1}(F)$-continuous sample paths in $\ell^{\infty}(\mathcal{G})$.

Proof. Doukhan and Surgailis (1998) proved that $\{\mathbb{G}_{n}(t),\ t\in\mathbb{R}\}$ converges weakly to a Gaussian process in the Skorohod space. Since $F$ is continuous under their assumptions, see Doukhan and Surgailis (1998, pp 88), the sample paths of limiting process of $\mathbb{G}_{n}$ are uniformly continuous with respect to the distance $d(s,t)=|F(s)-F(t)|$ based on the stationary distribution $F$. The proof now follows trivially from Theorem 1 and Lemma 3.

The recent papers by Dehling, Durieu and Volny (2009) and Dehling, Durieu and Tusche (2014) offer yet another, clever way to prove the weak limit of the standard empirical processes $\mathbb{G}_{n}$ based on stationary sequences that are not necessarily mixing. Their technique uses finite dimensional convergence coupled with a bound on the higher moments of partial sums, which in turn controls the dependence structure. Dehling, Durieu and Volny (2009) establishes the weak convergence of $\mathbb{G}_{n}$, while Dehling, Durieu and Tusche (2014) extends this idea to $\bar{\mathbb{Z}}_{n}$ indexed by more general classes of functions. However, these authors impose cumbersome entropy conditions and only manage to marginally extend the classes. For example, they prove weak convergence of the process $\int f_{t}\,\mathrm{d}\mathbb{G}_{n}$, indexed by functions $f_{t}(x)$ which constitute an one-dimensional monotone class (under the restrictive requirement that $\ s\leq t\Rightarrow f_{s}\leq f_{t}$). Theorem 1 applied in their setting, yields a more general result.

Corollary 5

Let $\mathcal{G}\subseteq BV_{T}$ and let $F$ be the stationary distribution of the sequence $\{X_{i}\}_{i\geq 1}$. Under assumptions (i) and (ii) in Section 1 of Dehling, Durieu and Volny (2009), $\{\int g\,{\rm d}\mathbb{G}_{n},\ g\in\mathcal{G}\}$ converges weakly to a Gaussian process with uniformly $L_{1}(F)$-continuous sample paths in $\ell^{\infty}(\mathcal{G})$.

Proof. The underlying distribution function $F$ of $X_{k}$ in Dehling, Durieu and Volny (2009) is continuous, see their display (3) at page 3702. Moreover, these authors establish weak convergence of $\mathbb{G}_{n}$ to a Gaussian process that has uniformly continuous sample paths with respect to the distance $d(s,t)=|F(s)-F(t)|$. The proof follows trivially from Theorem 1 and Lemma 3.

3 Bootstrap

The weak limit of $\bar{\mathbb{Z}}_{n}$ based on $\mathbb{G}_{n}$ in Theorem 1 is a Gaussian process $\bar{\mathbb{Z}}$ with complicated covariance structure

[TABLE]

for $f,g\in\mathcal{G}\subset BV_{T}$. A closed form solution is seldom available, so that actual applications of weak limit results of $\bar{\mathbb{Z}}_{n}$ are hard to implement. This situation calls for the bootstrap principle. Given the sample of first $n$ observations $\{X_{1},\ldots,X_{n}\}$, we let $\mathbb{G}_{n}^{\ast}$ be the bootstrap empirical process

[TABLE]

based on a bootstrap sample $\{X_{1,n}^{\ast},\ldots,X_{m_{n},n}^{\ast}\}$. We stress that no additional assumption on the structure of the variables $X_{i,n}^{\ast}$ is required for our purposes. Analogous to $\bar{\mathbb{Z}}_{n}(g)=\int g\,\mathrm{d}\mathbb{G}_{n}$, we define $\bar{\mathbb{Z}}_{n}^{\ast}(g):=\int g\,\mathrm{d}\mathbb{G}_{n}^{\ast}$ for any $g\in BV_{T}^{\prime}$ with $T<\infty$. We recall that the bounded Lipschitz distance

[TABLE]

between two processes $\mathbb{Z}_{n}^{\ast}$ and $\mathbb{Z}$ metrizes weak convergence. The symbol $\mathbb{E}^{\ast}$ stands for the expectation over the randomness of the bootstrap sample $X_{1,n}^{\ast},\ldots,X_{m_{n},n}^{\ast}$, conditionally given the original sample $X_{1},\ldots,X_{n}$, while $BL_{1}$ denotes the space of Lipschitz functionals $h:\ell^{\infty}(\mathcal{G})\rightarrow\mathbb{R}$ with $|h(x)|\leq 1$ and $|h(x)-h(y)|\leq\|x-y\|_{\infty}$ for all $x,y\in\mathcal{G}\subset BV_{T}$. As is customary in the literature, we speak of weak convergence in probability if the random variable $d_{BL}(\mathbb{Z}_{n}^{\ast},\mathbb{Z})$ converges to zero in probability; if it converges to zero almost surely, we speak of weak convergence almost surely.

Theorem 6

Let $\mathcal{G}\subseteq BV_{T}^{\prime}$. Assume that, conditionally on $X_{1},\ldots,X_{n}$, $\{\mathbb{G}_{n}^{\ast}(t),\ t\in\mathbb{R}\}$ converges weakly to a Gaussian process, in probability, that has unifomly continuous sample paths with respect to the distance $d(s,t)=|F(s)-F(t)|$ based on the stationary distribution $F$ of $X_{i}$. The following three statements hold true:

$\left\{\int g\,{\rm d}\mathbb{G}_{n}^{\ast},\ g\in\mathcal{G}\right\}$* converges weakly to a Gaussian process with uniformly $L_{1}(F)$-continuous sample paths in $\ell^{\infty}(\mathcal{G})$.* 2. 2.

If the weak convergence of $\{\mathbb{G}_{n}^{\ast}(t),\ t\in\mathbb{R}\}$ holds almost surely, then the conclusion that $\left\{\int g\,{\rm d}\mathbb{G}_{n}^{\ast},\ g\in\mathcal{G}\right\}$ converges weakly in $\ell^{\infty}(\mathcal{G})$ holds almost surely as well. 3. 3.

If $\{\mathbb{G}_{n}(t),\ t\in\mathbb{R}\}$ and $\{\mathbb{G}_{n}^{\ast}(t),\ t\in\mathbb{R}\}$ converge to the same limit, then so do $\left\{\int g\,{\rm d}\mathbb{G}_{n},\ g\in\mathcal{G}\right\}$ and $\left\{\int g\,{\rm d}\mathbb{G}_{n}^{\ast},\ g\in\mathcal{G}\right\}$.

The literature offers numerous bootstrapping techniques for stationary data, such as moving block bootstrap, stationary bootstrap, sieved bootstrap, Markov chain bootstrap, to name a few, but their validity is proved for specific cases/statistics only. Due to complications with entropy calculations for dependent triangular arrays, almost all results treat the standard empirical processes $\mathbb{G}_{n}^{\ast}$ with few notable exceptions. The moving block bootstrap was justified for VC-type classes, but only under rather restrictive $\beta$-mixing conditions on $X_{i}$ (Radulović, 1996). Bracketing classes were considered by Bühlmann (1995), but his conditions are even more restrictive. In contrast, the process $\{\mathbb{G}_{n}(t),$ $t\in\mathbb{R}\}$ is rather easy to bootstrap. This coupled with Theorem 6 offers the following result.

Corollary 7

Let $\{X_{j}\}_{j\geq 1}$ be a stationary sequence of random variables with continuous stationary distribution function $F$ and $\alpha$-mixing coefficients satisfying $\sum_{k\geq n}\alpha_{k}\leq Cn^{-\gamma}$, for some $0<\gamma<1/3$ and $C<\infty$. Let $\mathbb{G}_{n}^{\ast}$ be the bootstrapped standard empirical process based on the moving block bootstrap, with block sizes $b_{n}$, specified in Peligrad (1998, page 882). Then, for $\mathcal{G}\subset BV_{T}$, $\{\int g\,{\rm d}\mathbb{G}_{n}^{\ast},\ g\in\mathcal{G}\}$ converges weakly to Gaussian process, almost surely, with uniformly $L_{1}(F)$-continuous sample paths.

Proof. Theorem 2.3 of Peligrad (1998) establishes the convergence of $\mathbb{G}_{n}^{\ast}$ to a Gaussian process with uniformly continuous sample paths with respect to the distance $d(s,t)=|F(s)-F(t)|$ based on the stationary distribution $F$ of $X_{i}$. Invoke Theorem 6 and Lemma 3 to conclude the proof.

Just as for weak convergence of the empirical process based on stationary sequences, there are numerous results that consider bootstrap for stationary, non-mixing sequences. For example, El Ktaibi, Ivanoff and Weber (2014) study short memory causal linear sequences, and prove weak convergence of $\mathbb{G}_{n}^{\ast}$ under conditions (on the growth of the weights $a_{j}$, the characteristic function of $\xi_{0}$ and moments of $\xi_{0}$) akin to the ones required for its non-bootstrap counterpart $\mathbb{G}_{n}$ (Doukhan and Surgailis, 1998). Again, Theorem 6 easily extends their result.

Corollary 8

Let $\{X_{i}=\sum_{j\geq 0}a_{j}\xi_{i-j}\}_{i\geq 1}$ be a sequence of random variables with stationary distribution $F$ such that conditions of El Ktaibi, Ivanoff and Weber (2014) are satisfied. Then, for any $\mathcal{G}\subset BV_{T}$, $\{\int g\,{\rm d}\mathbb{G}_{n}^{\ast},\ g\in\mathcal{G}\}$ converges weakly to a Gaussian limit with uniformly $L_{1}(F)$-continuous sample paths, almost surely.

Proof. El Ktaibi, Ivanoff and Weber (2014) establish weak convergence of $\mathbb{G}_{n}^{\ast}$ in the Skorohod space, for continuous $F,$ which in turn implies that the limiting process has uniformly continuous sample paths with respect to distance $d(s,t)=|F(s)-F(t)|$. Again, invoke Theorem 6 and Lemma 3 to conclude the proof.

4 Proofs for Theorem 1 and Theorem 6

We first give a short proof of Theorem 1 if the limit $\mathbb{Z}(t)$ of $\mathbb{Z}_{n}(t)$ has continuous sample paths with respect to the Lebesgue measure. Let $d_{BL}$ be the bounded Lipschitz metric that metrizes weak convergence, see, e.g., Van der Vaart and Wellner (1996, page 73) for the definition. Set $\bar{\mathbb{Z}}_{n}(g)=\int g\,\mathrm{d}\mathbb{Z}_{n}$ for any $g\in BV_{T}^{\prime}$. Assumptions A1 and A2 imply that the Lebesgue-Stieltjes integrals

[TABLE]

are well defined. Next, by the integration by parts formula in Lemma A, we have

[TABLE]

with

[TABLE]

in probability, as $n\rightarrow\infty$, since $\mathbb{Z}$ has uniformly continuous sample paths. For fixed $g$, $\tilde{\mathbb{Z}}_{n}(g)$ converges weakly to $\tilde{\mathbb{Z}}(g):=-\int\mathbb{Z}\,\mathrm{d}g$ by the continuous mapping theorem and weak convergence of $\mathbb{Z}_{n}$. The continuous mapping theorem also guarantees that the limit $\tilde{\mathbb{Z}}$ is tight in $\ell^{\infty}(\mathcal{G})$ as the map $X\mapsto\Gamma_{g}(X)=-\int X\,\mathrm{d}g$, $g\in\mathcal{G}$, is continuous. By the triangle inequality,

[TABLE]

The second inequality follows since the map $\Gamma_{f}(X):=\int X\ \mathrm{d}f$ is linear and Lipschitz with Lipschitz constant $\int\,|\mathrm{d}f|\leq T$ and the suprema are taken over all Lipschitz functionals $H:\ell^{\infty}(\mathcal{G})\rightarrow\mathbb{R}$ with $\|H\|_{\infty}\leq 1$ and $|H(X)-H(Y)|\leq\|X-Y\|_{\infty}$ and $H^{\prime}:\ell^{\infty}(\mathbb{R})\rightarrow\mathbb{R}$ with $\|H^{\prime}\|_{\infty}\leq 1$ and $|H^{\prime}(X)-H^{\prime}(Y)|\leq\|X-Y\|_{\infty}$, respectively. Together with the tightness of the limit $\tilde{\mathbb{Z}}$, this implies that the empirical process $\bar{\mathbb{Z}}_{n}(g)$ indexed by $g\in\mathcal{G}\subset BV_{T}^{\prime}$ converges weakly.

The above proof for continuous limit processes $\mathbb{Z}$ (that is, processes $\mathbb{Z}$ with uniformly continuous sample paths) is rather simple. Nevertheless, we could not find an actual publication of this trick. We would like to stress that the extension of this proof to general, non-continuous processes, is not trivial. Technical complications related to the interplay between the atoms of the limiting process $\mathbb{Z}(t)$ and discontinuities of $g\in\mathcal{G},$ require some care.

Lemma 9

Assume that $\mathbb{Z}_{n}(t)$ converges weakly to a Gaussian process $\mathbb{Z}(t)$, as $n\rightarrow\infty$. Then, for any right continuous function $g:\mathbb{R}\rightarrow\mathbb{R}$ of bounded variation $\overline{\mathbb{Z}}_{n}(g)$=$\int g\,{\rm d}\mathbb{Z}_{n}$ is well defined, and converges to a normal distribution on $\mathbb{R}.$

Proof. Let $g$ be an arbitrary right-continuous function of bounded variation. First, we notice that $\int g\,{\rm d}\mathbb{Z}_{n}$ and $\int\mathbb{Z}_{n}\,{\rm d}g$ are indeed well defined as Lebesgue-Stieltjes integrals. Recall that $g$ can have only countably many discontinuities which we will denote $a_{i}.$ By the integration by parts formula, Lemma A in the appendix, we have

[TABLE]

with operators $T_{1},T_{2}:\ell^{\infty}(\mathbb{R})\rightarrow\mathbb{R}$

[TABLE]

Since $g$ has finite variation, it is bounded with $\sum_{i}|\alpha_{i}|<\infty$ for the jumps $\alpha_{i}:=g(a_{i})-g(a_{i}^{-})$. Hence, the operator $T_{2}$, given by

[TABLE]

is linear and continuous. We conclude the proof by observing that the continuity of the operators $T_{1}$ and $T_{2}$ and the weak convergence of $\mathbb{Z}_{n}(t)$ to a Gaussian process $\mathbb{Z}$ ensures that these sequences $Y_{n}:=\int g\,{\rm d}\mathbb{Z}_{n}=T_{1}(\mathbb{Z}_{n})+T_{2}(\mathbb{Z}_{n})$ converge weakly to a normal distribution via the continuous mapping theorem.

For any distribution function $F_{0}$ and any $\beta>0$, we define

[TABLE]

with the convention that the supremum taken over the empty set is equal to zero.

Clearly, ${\mathbb{g}}_{\beta,F_{0}}(\mathbb{Z}_{n})=0$ for all $\beta>0$ if $F_{0}$ is continuous. In general, for arbitrary $F_{0}$, the quantity ${\mathbb{g}}_{\beta,F_{0}}(\mathbb{Z}_{n})$ is bounded in probability, for all $\beta>0$, by the continuous mapping theorem, as long as $\mathbb{Z}_{n}$ converges weakly. The following lemma plays an instrumental role and it could be perhaps of independent interest.

Lemma 10** (Decoupling Lemma)**

For any distribution function $F_{0}$ on $\mathbb{R}$, any right-continuous function $g:\mathbb{R}\rightarrow\mathbb{R}$ and any $\beta>0$, we have

[TABLE]

Here $\|g\|_{L_{1}(F_{0})}=\int|g|\,\,{\rm d}F_{0}$ and $\mathbb{Z}_{n}$ satisfies assumptions A1 and A2.

Proof. Without loss of generality we can assume that $||g||_{L_{1}(F_{0})}+||g||_{TV}<\infty.$ Since $F_{0}$ is a distribution function we can construct, for any $0<\beta<1,$ a finite grid $-\infty=s_{0}<s_{1}<...<s_{M}<\infty$ such that

[TABLE]

and

[TABLE]

leaving the possible jumps $F_{0}(s_{j})-F_{0}(s_{j}^{-})$ unspecified. Based on this grid we approximate $\mathbb{Z}_{n}(t)$ by

[TABLE]

and we set $\widetilde{\mathbb{Z}}_{n}(\pm\infty)=0.$ We observe that by construction

[TABLE]

Since the process $\widetilde{\mathbb{Z}}_{n}$ inherits the bounded variation properties of $\mathbb{Z}_{n},$

[TABLE]

is well defined for any right-continuous function $g$ of bounded variation. Using the integration by parts formula (Lemma A in the appendix) we obtain for the last term on the right

[TABLE]

Since $g$ is of bounded variation, it has countably many discontinuities $a_{i}$. Using (3), we obtain

[TABLE]

Consequently,

[TABLE]

Next we deal with the finite dimensional approximation

[TABLE]

Clearly,

[TABLE]

For the first term in (4) we introduce the step function

[TABLE]

designed to approximate $|g(t)|.$ Clearly $0\leq g^{\ast}(t)\leq|g(t)|$ and

[TABLE]

Hence, by (3) we have

[TABLE]

For the second term in (4), we have

[TABLE]

using for the last inequality

[TABLE]

and

[TABLE]

Lemma 10 now follows by combining the estimates (3) and (4) and (5).

An immediate corollary is the following result.

Corollary 11

For any distribution function $F_{0}$ and for all $T<\infty$, $\delta>0$ and $p\geq 1$, we have

[TABLE]

where the $\sup$ is taken over all right-continuous functions $g$ with $\|g\|_{TV}\leq T$ and $\|g\|_{L_{p}(F_{0})}=\left(\int|g|^{p}\mathrm{d}F_{0}\right)^{1/p}\leq\delta$.

Proof. The proof follows trivially from Lemma 10 by taking $\beta=\sqrt{\delta}$ and observing that $\|g\|_{L_{1}(F_{0})}\leq\|g\|_{L_{p}(F_{0})}$ for $p\geq 1.$

Proof of Theorem 1.

First we recall, see, for instance, Chapter 1.5 in Van der Vaart and Wellner (1996), that $\{\bar{\mathbb{Z}}_{n}(g)$, $g\in\mathcal{G}\}$ converges weakly to a tight limit in $\ell^{\infty}(\mathcal{G})$, provided

• (a)

the marginals $(\bar{\mathbb{Z}}_{n}(g_{1}),\ldots,\bar{\mathbb{Z}}_{n}(g_{k}))$ converge weakly for every finite subset $g_{1},\ldots,g_{k}\in\mathcal{G}$, and

• (b)

there exists a semi-metric $\rho$ on $\mathcal{G}$ such that $(\mathcal{G},\rho)$ is totally bounded and $\bar{\mathbb{Z}}_{n}(g)$ is $\rho$-stochastically equicontinuous, that is,

[TABLE]

for all $\varepsilon>0$.

The finite dimensional convergence (a) follows trivially from Lemma 9, linearity of the process $\overline{\mathbb{Z}}_{n}(g)$ and the Cramèr-Wold device. As for the stochastic equicontinuity (b) of $\overline{Z}_{n}(g)$, it is sufficient to show that, for every $\varepsilon>0,$

[TABLE]

where the supremum is taken over all differences $h=g-g{{}^{\prime}}$ with $g,g^{\prime}\in\mathcal{G}$ and $\left\|h\right\|_{L_{1}(F_{0})}=\int|h|dF_{0}\leq\delta.$ Since $h$ is also right-continuous and $\left\|h\right\|_{TV}\leq 2T$, Corollary 11 implies that

[TABLE]

Let $f_{t}(x)=1\{x\leq t\},$ so that $\mathbb{Z}_{n}(t)=\overline{\mathbb{Z}}_{n}(f_{t})$ and

[TABLE]

and observe that

[TABLE]

The weak convergence of $\mathbb{Z}_{n}(t)$, or equivalently, $\overline{\mathbb{Z}}_{n}(f_{t})$, to a continuous (with respect to $d(.,.)$) process $\mathbb{Z}(t)$ implies that

[TABLE]

Moreover, the weak convergence of $\mathbb{Z}_{n}(t)$ implies that ${\mathbb{g}}_{\sqrt{\delta},F_{0}}(\mathbb{Z}_{n})$ is bounded in probability, so ${\mathbb{g}}_{\sqrt{\delta},F_{0}}(\mathbb{Z}_{n})\sqrt{\delta}\rightarrow 0$ as $\delta\rightarrow 0$ and $n\rightarrow\infty.$

Summarizing, $\bar{\mathbb{Z}}_{n}(g),$ converges for each $g\in\mathcal{G}$ to a Gaussian limit and $\bar{\mathbb{Z}}_{n}$ is uniformly $L_{1}(F_{0})$-equicontinuous, in probability. Moreover, $(\mathcal{G},L_{1}(F_{0}))$ is totally bounded (for any distribution $F_{0})$. This well-known fact can be found in Example 2.6.21 in Van der Vaart and Wellner (1996, page 149). It implies the weak convergence of $\bar{\mathbb{Z}}_{n}(g)$ to a process with uniformly $L_{1}(F_{0})-$continuous sample paths (see Theorems 1.5.4 and 1.5.7 in Van der Vaart and Wellner 1996).

Proof of Theorem 6.

Here we only prove the “in probability” case. The almost sure statement follows after straightforward changes in the proof of Theorem 1. A simple modification of Lemma 9 yields

[TABLE]

for the same operators $T_{1}$ and $T_{2}$ as defined in Lemma 9. Since $\mathbb{G}_{n}^{\ast}$ converges to a Gaussian limit the finite dimensional convergence follows by repeating the computation presented in the proof of Lemma 9 after replacing $\mathbb{G}_{n}(t)$ with $\mathbb{G}_{n}^{\ast}(t)$. As for stochastic equicontinuity of $\mathbb{Z}_{n}^{\ast}(g)$ we find, analogous to Corollary 11, that for $\delta>0$

[TABLE]

Now the weak convergence of $\mathbb{Z}_{n}^{\ast}$ follows from the convergence of $\mathbb{G}_{n}^{\ast}$, as in the proof of Theorem 1, conditionally given the sample $X_{1},\ldots,X_{n}$. Moreover, if $\mathbb{G}_{n}(t)$ and $\mathbb{G}_{n}^{\ast}(t)$ converge to the same Gaussian process, then Lemma 9 coupled with the Cramèr-Wold device implies that the finite dimensional distribution of the limiting process of $\mathbb{Z}_{n}(g)$ and $\mathbb{Z}_{n}^{\ast}(g)$ are the same. This concludes the proof.

Appendix A Appendix

For completeness, we state the following classical result and give a simple elementary proof which was communicated to us by David Pollard. Theorem 18.4 in Billingsley (1986) states the result for functions on a bounded interval, yet its proof can be easily extended to the entire real line.

Lemma A. Let $f$ and $g$ be right-continuous functions of bounded variation and define measures $\mu$ and $\nu$ as $\mu(-\infty,x]=f(x)-f(-\infty)$ and $\nu(-\infty,y]=g(y)-g(-\infty)$. Then

[TABLE]

Moreover, if either $f(\pm\infty)=0$ or $g(\pm\infty)=0$, then

[TABLE]

Proof. Set $H(x,y)=1\{x\leq y\}$ and observe that by the very definition of Lebesgue integral

[TABLE]

and

[TABLE]

Hence

[TABLE]

Next we apply Fubini

[TABLE]

to prove the lemma.

Acknowledgements.

We would like to thank both referees for their careful reading and many constructive remarks.

The research of Wegkamp was supported in part by NSF grants DMS-1310119 and DMS-1712709.