Qualitative robustness for bootstrap approximations

TL;DR

This paper investigates the qualitative robustness of bootstrap approximations for non-i.i.d. data, extending existing theory to dependent processes like $eta$-mixing, and establishes conditions under which robustness is preserved.

Contribution

It extends the theory of qualitative robustness of bootstrap methods to dependent data processes, introducing new conditions for robustness beyond i.i.d. assumptions.

Findings

Qualitative robustness holds for certain dependent processes under specific conditions.

Continuity of the statistical operator is crucial for robustness.

A convergence condition called the Varadarajan property is necessary for robustness in dependent data.

Abstract

An important property of statistical estimators is qualitative robustness, that is small changes in the distribution of the data only result in small chances of the distribution of the estimator. Moreover, in practice, the distribution of the data is commonly unknown, therefore bootstrap approximations can be used to approximate the distribution of the estimator. Hence qualitative robustness of the statistical estimator under the bootstrap approximation is a desirable property. Currently most theoretical investigations on qualitative robustness assume independent and identically distributed pairs of random variables. However, in practice this assumption is not fulfilled. Therefore, we examine the qualitative robustness of bootstrap approximations for non-i.i.d. random variables, for example $\alpha$-mixing and weakly dependent processes. In the i.i.d. case qualitative robustness is ensured via the continuity of the statistical operator, representing the estimator, see Hampel (1971) and Cuevas and Romo (1993). We show, that qualitative robustness of the bootstrap approximation is still ensured under the assumption that the statistical operator is continuous and under an additional assumption on the stochastic process. In particular, we require a convergence condition of the empirical measure of the underlying process, the so called Varadarajan property.

Equations

$$\displaystyle S\big{(}\mathbb{P}_{\mathbf{w}_{n}}\big{)}\;=\;S_{n}(\mathbf{w}_{n})\;=\;S_{n}(z_{1},\dots,z_{n})\qquad\forall\,\mathbf{w}_{n}=(z_{1},\dots,z_{n})\in\mathcal{Z}^{n},\quad\forall\,n\in\mathbb{N},$$

$$d_{\text{BL}}(P,Q):=\sup \left\{\left|\int fdP-\int fdQ\right|;f\in \text{BL}(\mathcal{X}),\Vert f{\Vert }_{\text{BL}}\le 1\right\}$$

$$d_{\text{BL}}(P_n,Q_n)<\delta \Rightarrow d_{\text{BL}}(\mathcal{L}({\mathcal{L}}_{P_n^{\ast }}(S_n)),\mathcal{L}({\mathcal{L}}_{Q_n^{\ast }}(S_n)))<\epsilon .$$

$$\pi ({\mathbb{P}}_{{\mathbf{W}}_n},P)\underset{n\to \infty }{\to }0\text{almost surely.}$$

$$\pi ({\mathbb{P}}_{{\mathbf{W}}_n},P)\underset{n\to \infty }{\to }0\text{in probability.}$$

$$d_{n,p}\big{(}(z_{1},\dots,z_{n}),(z_{1}^{\prime},\dots,z_{n}^{\prime})\big{)}\;=\;\big{\|}\big{(}d_{\mathcal{Z}}(z_{1},z_{1}^{\prime}),\dots,d_{\mathcal{Z}}(z_{n},z_{n}^{\prime})\big{)}\big{\|}_{p}$$

$$d_{n}\big{(}(z_{1},\dots,z_{n}),(z_{1}^{\prime},\dots,z_{n}^{\prime})\big{)}\;=\;\inf\big{\{}\varepsilon>0:\sharp\{i:d(z_{i},z_{i}^{\prime})\geq\varepsilon\}/n\leq\varepsilon\big{\}}\,.$$

$$P^i=(1-{\epsilon }_i)P+\epsilon {\tilde{P}}^i$$

$$d_{\text{BL}}(P_n,Q_n)<\delta \Rightarrow d_{\text{BL}}(\mathcal{L}({\mathcal{L}}_{P_n^{\ast }}(S_n)),\mathcal{L}({\mathcal{L}}_{Q_n^{\ast }}(S_n)))<\epsilon .$$

$$\alpha ({\mathcal{A}}_1,{\mathcal{A}}_2,\mu ):=\sup \{|\mu (A_1\cap A_2)-\mu (A_2)\mu (A_2)||A_1\in {\mathcal{A}}_1,A_2\in {\mathcal{A}}_2\}.$$

$$\alpha ((Z{)}_{i\in \mathbb{N}},\mu ,i,j)=\alpha (\sigma (Z_i),\sigma (Z_j),\mu )$$

$$\alpha ((Z{)}_{i\in \mathbb{N}},\mu ,n)=\underset{i\ge 1}{\sup }\alpha (\sigma (Z_i),\sigma (Z_{i+n}),\mu ).$$

$$\underset{n\to \infty }{\lim }\alpha ((Z{)}_{i\in \mathbb{N}},\mu ,n)=0.$$

$$\underset{n\to \infty }{\lim }\frac{1}{n^2}\sum _{i=1}^n\sum _{j=1}^n\alpha ((Z{)}_{i\in \mathbb{N}},\mu ,i,j)=0.$$

$$n^h\in \mathcal{O}(b(n)),b(n)\in \mathcal{O}(n^{1/3-a}),\text{for some}0<h<\frac{1}{3}-a,0<a<\frac{1}{3},$$

$$\sum _{m>n}\alpha (\sigma (Z_1,\dots ,Z_i),\sigma (Z_{i+m},\dots ),P_{\mathbb{N}})=\mathcal{O}(n^{-\gamma }),i\in \mathbb{N},\text{for some}\gamma >0.$$

$$b(n)=\mathcal{O}(n^{\frac{1}{2}-a})\text{for some}a>0.$$

$$\sum _{m=0}^{\infty }(m+1{)}^{8d+7}(\alpha (\sigma (Z_1,\dots ,Z_i),\sigma (Z_{i+m},\dots ),P_{\mathbb{N}}){)}^{\frac{1}{2}}<\infty ,i\in \mathbb{N}.$$

$$d_{\text{BL}}(P_n,Q_n)\le \delta \Rightarrow d_{\text{BL}}\left(\frac{1}{n}\sum _{i=1}^n{P}^i,\frac{1}{n}\sum _{i=1}^n{Q}^i\right)\le \delta .$$

$${\int }_{\mathcal{Z}}f(z_i)d{P}^i(z_i)={\int }_{{\mathcal{Z}}^{n-1}}{\int }_{\mathcal{Z}}f(z_i)d{P}^i(z_i)d\left({\otimes }_{j\ne i}{P}^j(z_j)\right).$$

$$\underset{f\in {\text{BL}}_1(\mathcal{Z})}{\sup }$$

$$=\underset{f\in {\text{BL}}_1(\mathcal{Z})}{\sup }\left|\frac{1}{n}\sum _{i=1}^n\left[{\int }_{\mathcal{Z}}f(z_i)d{P}^i(z_i)-{\int }_{\mathcal{Z}}f(z_i)d{Q}^i(z_i)\right]\right|$$

$$\stackrel{\text{eqref}proof:qualitativerobustnessboottsrapLemmaproduktmassundMischmass}{=}\underset{f\in {\text{BL}}_1(\mathcal{Z})}{\sup }|\frac{1}{n}\sum _{i=1}^n[{\int }_{{\mathcal{Z}}^{n-1}}{\int }_{\mathcal{Z}}f(z_i)d{P}^i(z_i)d\left({\otimes }_{j\ne i}{P}^j(z_j)\right)$$

$$-{\int }_{{\mathcal{Z}}^{n-1}}{\int }_{\mathcal{Z}}f(z_i)d{Q}^i(z_i)d\left({\otimes }_{j\ne i}{Q}^j(z_j)\right)]|$$

$$=\underset{f\in {\text{BL}}_1(\mathcal{Z})}{\sup }\left|\frac{1}{n}\sum _{i=1}^n\left[{\int }_{{\mathcal{Z}}^n}f(z_i)d\left({\otimes }_{j=1}^n{P}^j(z_j)\right)-{\int }_{{\mathcal{Z}}^n}f(z_i)d\left({\otimes }_{j=1}^n{Q}^j(z_j)\right)\right]\right|$$

$$\le \frac{1}{n}\sum _{i=1}^n\underset{f\in {\text{BL}}_1(\mathcal{Z})}{\sup }\left|{\int }_{{\mathcal{Z}}^n}f(z_i)d\left({\otimes }_{j=1}^n{P}^j(z_j)\right)-{\int }_{{\mathcal{Z}}^n}f(z_i)d\left({\otimes }_{j=1}^n{Q}^j(z_j)\right)\right|.$$

$$|\tilde{f}(z_1,\dots ,z_n)-$$

$$\le |f{|}_{1}d(z_i,z_i^{\prime })\le |f{|}_1(d_{\mathcal{Z}}(z_1,z_1^{\prime })+\dots +d_{\mathcal{Z}}(z_i,z_i^{\prime })+\dots +d_{\mathcal{Z}}(z_n,z_n^{\prime })),$$

$$d_{\text{BL}}\left(\frac{1}{n}\sum _{i=1}^n{P}^i,\frac{1}{n}\sum _{i=1}^n{Q}^i\right)$$

$$\le \frac{1}{n}\sum _{i=1}^n{d}_{\text{BL}}\left(P_n,Q_n\right)\le \delta ,$$

$$d_{\text{BL}}({\mathcal{L}}_{P_n^{\ast }}(S_n),{\mathcal{L}}_{Q_n^{\ast }}(S_n))$$

$${\mathcal{L}}_{P_n}(S_n)=S_n(P_n)=S_n\circ {\mathbf{W}}_n(P_{\mathbb{N}})\text{and}{\mathcal{L}}_{P_n^{\ast }}(S_n)=S_n(P_n^{\ast })=S_n\circ {\mathbf{W}}_n^{\prime }(P_{\mathbb{N}}^{\ast })$$

$$d_{\text{BL}}({\mathcal{L}}_{P_n^{\ast }}(S_n),{\mathcal{L}}_{P_n}(S_n))$$

$$\underset{n\to \infty }{\lim }\underset{P\in \mathcal{M}(\mathcal{Z})}{\sup }{P}_{\mathbb{N}}\left(\left\{z_{\mathbb{N}}\in {\mathcal{Z}}^{\mathbb{N}}|\underset{m\ge n}{\sup }{d}_{\text{BL}}({\mathbb{P}}_{{\mathbf{W}}_m(z_{\mathbb{N}})},P)>\eta \right\}\right)=0.$$

Full text

Keywords: stochastic processes, qualitative robustness, bootstrap, $\alpha$-mixing, weakly dependent AMS: 60G20, 62G08, 62G09, 62G35

1 Introduction

The overwhelming part of theoretical publications in statistical machine learning was done under the assumption that the data is generated by independent and identically distributed (i.i.d.) random variables. However, this assumption is not fulfilled in many practical applications so that non-i.i.d. cases increasingly attract attention in machine learning. An important property of an estimator is robustness. It is well known that many classical estimators are not robust, which means that small changes in the distribution of the data generating process may highly affect the results, see for example Huber (1981), Hampel (1968), Jurečková and Picek (2006) or Maronna et al. (2006) for some books on robust statistics. Qualitative robustness is a continuity property of the estimator and means roughly speaking: small changes in the distribution of the data only lead to small changes in the distribution (i.e. the performance) of the estimator. In this way the following kinds of "small errors" are covered: small errors in all data points (rounding errors) and large errors in only a small fraction of the data points (gross errors, outliers). Qualitative robustness of estimators has been defined originally in Hampel (1968) and Hampel (1971) in the i.i.d. case and has been generalized to estimators for stochastic processes in various ways, for example, in Papantoni-Kazakos and Gray (1979), Bustos (1980), which will be the one used here, Cox (1981), Boente et al. (1987), Zähle (2015), and Zähle (2016), for a more local consideration of qualitative robustness, see for example Krätschmer et al. (2017).

Often the finite sample distribution of the estimator or of the stochastic process of interest is unknown, hence an approximation of the distribution is needed. Commonly, the bootstrap is used to receive an approximation of the unknown finite sample distribution by resampling from the given sample.

The classical bootstrap, also called the empirical bootstrap, has been introduced by Efron (1979) for i.i.d. random variables. This concept is based on drawing a bootstrap sample $(Z_{1}^{*},\ldots,Z_{m}^{*})$ of size $m\in\mathbb{N}$ with replacement out of the original sample $(Z_{1},\ldots,Z_{n})$, $n\in\mathbb{N}$, and approximate the theoretical distribution $P_{n}$ of $(Z_{1},\ldots,Z_{n})$ using the bootstrap sample. For the empirical bootstrap the approximation of the distribution via the bootstrap is given by the empirical distribution of the bootstrap sample $(Z^{*}_{1},\ldots,Z^{*}_{m})$, hence $P^{*}_{n}=\otimes_{i=1}^{n}\left(\frac{1}{m}\sum_{i=1}^{m}\delta_{Z_{i}^{*}}\right)$, where $\delta_{Z_{i}}$ denotes the dirac measure. The bootstrap sample itself has distribution $\otimes_{i=1}^{m}\left(\frac{1}{n}\sum_{i=1}^{n}\delta_{Z_{i}}\right)$.

For an introduction to the bootstrap see for example Efron and Tibshirani (1993) and van der Vaart (1998, Chapter 3.6). Besides the empirical bootstrap many other bootstrap methods have been developed in order to find good approximations also for non-i.i.d. observations, see for example Singh (1981), Lahiri (2003), and the references therein. In Section 2.2 the moving block bootstrap introduced by Künsch (1989) and Liu and Singh (1992) is used to approximate the distribution of an $\alpha$-mixing stochastic process.

It is, also in the non-i.i.d. case, still desirable that the estimator is qualitatively robust even for the bootstrap approximation. That is, the distribution of the estimator under the bootstrap approximation $\mathcal{L}_{P_{n}^{*}}(S_{n})$, $n\in\mathbb{N}$, of the assumed, ideal distribution $P_{n}$ should still be close to the distribution of the estimator under the bootstrap approximation $\mathcal{L}_{Q_{n}^{*}}(S_{n})$, $n\in\mathbb{N}$, of the real contaminated distribution $Q_{n}$. Remember that this is a random object as $P_{n}^{*}$ respectively $Q^{*}_{n}$ are random. For notational convenience all bootstrap values are noted as usual with an asterisk.

To show qualitative robustness often generalizations of Hampel’s theorem are used, as it is often hard to show qualitative robustness directly. For the i.i.d. case Hampel’s Theorem ensures qualitative robustness of a sequence of estimators, if these estimators are continuous and can be represented by a statistical operator which is continuous in the distribution of the data generating stochastic process. Accordingly we try to find results similar to Hampel’s theorem for the case of bootstrap approximations for non-i.i.d. cases.

Generalizations of Hampel’s theorem to non-i.i.d. cases can be found in Zähle (2015) and Zähle (2016). For a slightly different generalization of qualitative robustness, Hampel’s theorem has been formulated for strongly stationary and ergodic processes in Cox (1981) and Boente et al. (1982). In Strohriegl and Hable (2016) a generalization of Hampel’s Theorem to a broad class of non-i.i.d. stochastic processes is given. Cuevas and Romo (1993) describes a concept of qualitative robustness of bootstrap approximations for the i.i.d. case and for real valued estimators. Also a generalization of Hampel’s theorem to this case is given. In Christmann et al. (2013, 2011) qualitative robustness of Efron’s bootstrap approximation is shown for the i.i.d. case for a class of regularized kernel based learning methods, i. e. not necessarily real valued estimators. Moreover Beutner and Zähle (2016) describes consistency of the bootstrap for plug in estimators.

The next chapter contains a definition of qualitative robustness of the bootstrap approximation of an estimator and the main results. In Chapter 2.1 Theorem 2.2 shows qualitative robustness of the bootstrap approximation of an estimator for independent but not necessarily identically distributed random variables, Chapter 2.2 contains Theorem 2.6 and 2.7 which generalize the result in Christmann et al. (2013) to $\alpha$-mixing sequences with values in $\mathbb{R}^{d}$. All proofs are deferred to the appendix.

2 Qualitative robustness for bootstrap estimators

Throughout this paper, let $(\mathcal{Z},d_{\mathcal{Z}})$ be a Polish space with some metric $d_{\mathcal{Z}}$ and Borel-$\sigma$-algebra $\mathcal{B}$. Denote by $\mathcal{M}(\mathcal{Z}^{\mathbb{N}})$ the set of all probability measures on $(\mathcal{Z}^{\mathbb{N}},\mathcal{B}^{\otimes\mathbb{N}})$. Let $(\mathcal{Z}^{\mathbb{N}},\mathcal{B}^{\otimes\mathbb{N}},\mathcal{M}(\mathcal{Z}^{\mathbb{N}}))$ be the underlying statistical model. If nothing else is stated, we always use Borel-$\sigma$-algebras for topological spaces. Let $(Z_{i})_{i\in\mathbb{N}}$ be the coordinate process on $\mathcal{Z}^{\mathbb{N}}$, that is $Z_{i}:\mathcal{Z}^{\mathbb{N}}\rightarrow\mathcal{Z},\;(z_{j})_{j\in\mathbb{N}}\mapsto z_{i}$, $i\in\mathbb{N}$. Then the process has law $P_{\mathbb{N}}$ under $P_{\mathbb{N}}\in\mathcal{M}(\mathcal{Z}^{\mathbb{N}})$. Moreover let $P_{n}:=(Z_{1},\ldots,Z_{n})\circ P_{\mathbb{N}}$ be the $n$-th order marginal distribution of $P_{\mathbb{N}}$ for every $n\in\mathbb{N}$ and $P_{\mathbb{N}}\in\mathcal{M}(\mathcal{Z}^{\mathbb{N}})$. We are concerned with a sequence of estimators $(S_{n})_{n\in\mathbb{N}}$ on the stochastic process $(Z_{i})_{i\in\mathbb{N}}$. The estimator may take its values in any Polish space $H$ with some metric $d_{H}$; that is, $S_{n}:\mathcal{Z}^{n}\rightarrow H$ for every $n\in\mathbb{N}$.

Our work applies to estimators which can be represented by a statistical operator $S:\mathcal{M}(\mathcal{Z})\rightarrow H$, that is,

[TABLE]

where $\mathbb{P}_{\mathbf{w}_{n}}$ denotes the empirical measure defined by $\mathbb{P}_{\mathbf{w}_{n}}(B):=\frac{1}{n}\sum_{i=1}^{n}I_{B}(z_{i})$, $B\in\mathcal{B}$, for the observations $\mathbf{w}_{n}=(z_{1},...,z_{n})\in\mathcal{Z}^{n}$. Examples of such estimators are M-estimators, R-estimators, see Huber (1981, Theorem 2.6), or Support Vector Machines, see Hable and Christmann (2011).

Based on the generalization of Hampel’s concept of $\Pi$-robustness from Bustos (1980), we define qualitative robustness for bootstrap approximations for non-i.i.d sequences of random variables. The stronger concept of $\Pi$-robustness is needed here, as we do not assume to have i.i.d. random variables, which are used in Cuevas and Romo (1993).

Therefore the definition of qualitative robustness stated below is stronger than the definition in Cuevas and Romo (1993), i. e. if we use this definition for the i.i.d. case the assumption $d_{\textup{BL}}(P_{n},Q_{n})=d_{\textup{BL}}(\otimes_{i=1}^{n}P,\otimes_{i=1}^{n}Q)<\delta$ implies $d_{\textup{BL}}(P,Q)<\delta$, where $d_{\textup{BL}}$ denotes the bounded Lipschitz metric. This can be seen similar to the proof of Lemma 3.1 in Section 2.1.

Now, let $P_{\mathbb{N}}^{*}$ be the approximation of $P_{\mathbb{N}}$ with respect to the bootstrap. Define the bootstrap sample $(Z^{*}_{1},\ldots,Z^{*}_{n})$ as the first $n$ coordinate projections $Z^{*}_{i}:\mathcal{Z}^{\mathbb{N}}\rightarrow\mathcal{Z}$, where the law of the stochastic process $(Z^{*}_{i})_{i\in\mathbb{N}}$ has to be chosen according to the bootstrap procedure. For the empirical bootstrap, for example, the bootstrap sample is chosen via drawing with replacement from the given observations $z_{1},\ldots,z_{\ell}$, $\ell\in\mathbb{N}$. Hence the distribution of the bootstrap sample is $\otimes_{n\in\mathbb{N}}\frac{1}{\ell}\sum_{i=1}^{\ell}\delta_{z_{i}}$, with finite sample distributions $\otimes_{j=1}^{n}\frac{1}{\ell}\sum_{i=1}^{\ell}\delta_{z_{i}}=(Z^{*}_{1},\ldots,Z^{*}_{n})\left(\otimes_{n\in\mathbb{N}}\frac{1}{\ell}\sum_{i=1}^{\ell}\delta_{z_{i}}\right)$.

Contrarily to the classical case of qualitative robustness the distribution of the estimator under $P^{*}_{n}$, $\mathcal{L}_{P^{*}_{n}}(S_{n})$ is a random probability measure, as the distribution $P^{*}_{n}=\otimes_{i=1}^{n}\frac{1}{\ell}\sum_{i=1}^{\ell}\delta_{Z^{*}_{i}}$, $Z^{*}_{i}:\mathcal{Z}^{\mathbb{N}}\rightarrow\mathcal{Z}$, is random. Hence the mapping $z_{\mathbb{N}}\mapsto\mathcal{L}_{P^{*}_{n}}(S_{n})$, $z_{\mathbb{N}}\in\mathcal{Z}^{\mathbb{N}}$, is itself a random variable with values in $\mathcal{M}(H)$, i. e. on the space of probability measures on $H$, equipped with the weak topology on $\mathcal{M}(H)$. The measurability of this mapping is ensured by Beutner and Zähle (2016, Lemma D1).

Contrarily to the original definitions of qualitative robustness in Bustos (1980) the bounded Lipschitz metric $d_{\textup{BL}}$ is used instead of the Prohorov metric $\pi$ for the definition of qualitative robustness of the bootstrap approximation below. This is equivalent to Cuevas and Romo (1993). Let $\mathcal{X}$ be a separable metric space, then the bounded Lipschitz metric on the space of probability measures $\mathcal{M}(\mathcal{X})$ on $\mathcal{X}$ is defined by:

[TABLE]

where $\|\cdot\|_{\textup{BL}}:=|\cdot|_{1}+\|\cdot\|_{\infty}$ denotes the bounded Lipschitz norm with $|f|_{1}=\sup_{x\neq y}\frac{|f(x)-f(y)|}{d(x,y)}$ and $\|\cdot\|_{\infty}$ the supremum norm $\|f\|_{\infty}:=\sup_{x}|f(x)|$ and the space of bounded Lipschitz functions is defined as $\textup{BL}:=\{f:\mathcal{X}\rightarrow\mathbb{R}\;|\;f\;\text{Lipshitz and }\|f\|_{\textup{BL}}<\infty\}$. This is due to technical reasons only. Both metrics metricize the weak topology on the space of all probability measures $\mathcal{M}(\mathcal{X})$, for Polish spaces $\mathcal{X}$, see, for example, Huber (1981, Chapter 2, Corollary 4.3) or Dudley (1989, Theorem 11.3.3), and therefore can be replaced while adapting $\delta$ on the left hand-side of implication (2). If $\mathcal{X}$ is a Polish space, so is $\mathcal{M}(\mathcal{X})$ with respect to the weak topology, see Huber (1981, Chapter 2, Theorem 3.9). Hence the bounded Lipschitz metric on the right-hand side of implication (2) operates on a space of probability measures on the Polish space $\mathcal{M}(\mathcal{X})$. Therefore the Prohorov metric and the bounded Lipschitz metric can again be replaced while adapting $\varepsilon$ in (2). Similar to Cuevas and Romo (1993) the proof of the theorems below rely on the fact that the set of bounded Lipschitz functions BL is a uniform Glivenko-Cantelli class, which implies uniform convergence of the bounded Lipschitz metric of the empirical measure to a limiting distribution, see Dudley et al. (1991). Therefore the definition is given with respect to the bounded Lipschitz metric.

Definition 2.1** (Qualitative robustness for bootstrap approximations)**

Let $P_{\mathbb{N}}\in\mathcal{M}(\mathcal{Z}^{\mathbb{N}})$ and let $P^{*}_{\mathbb{N}}\in\mathcal{M}(\mathcal{Z}^{\mathbb{N}})$ be the bootstrap approximation of $P_{\mathbb{N}}$. Let $\mathcal{P}\subset\mathcal{M}(\mathcal{Z}^{\mathbb{N}})$ with $P_{\mathbb{N}}\in\mathcal{P}$. Let $S_{n}:\mathcal{Z}^{n}\rightarrow H$, $n\in\mathbb{N}$, be a sequence of estimators. Then the sequence of bootstrap approximations $(\mathcal{L}_{P_{n}^{*}}(S_{n}))_{n\in\mathbb{N}}$ is called qualitatively robust at $P_{\mathbb{N}}$ with respect to $\mathcal{P}$ if, for every $\varepsilon>0$, there is $\delta>0$ such that there is $n_{0}\in\mathbb{N}$ such that for every $n\geq n_{0}$ and for every $Q_{\mathbb{N}}\in\mathcal{P}$,

[TABLE]

Here $\mathcal{L}(\mathcal{L}_{P_{n}^{*}}(S_{n}))$ (respectively $\mathcal{L}(\mathcal{L}_{Q_{n}^{*}}(S_{n}))$) denotes the distribution of the bootstrap approximation of the estimator $S_{n}$ under $P^{*}_{n}$ (respectively $Q^{*}_{n}$).

This definition of qualitative robustness with respect to the subset $\mathcal{P}$ indicates that we do not show (2) for arbitrary probability measures $Q_{\mathbb{N}}\in\mathcal{M}(\mathcal{Z}^{\mathbb{N}})$. All of our results require the contaminated process to at least have the same structure as the ideal process. This is due to the use of the bootstrap procedure. The empirical bootstrap, which is used below, only works well for a few processes, see for example Lahiri (2003), hence the assumptions on the contaminated process are necessary. To our best knowledge there are no results concerning qualitative robustness of the bootstrap approximation for general stochastic processes without any assumptions on the second process and it is probably very hard to show this for every $Q_{\mathbb{N}}\in\mathcal{M}(\mathcal{Z}^{\mathbb{N}})$, respectively $\mathcal{P}=\mathcal{M}(\mathcal{Z}^{\mathbb{N}})$. Another difference to the classical definition of qualitative robustness in Bustos (1980) is the restriction to $n\geq n_{0}$. As the results for the bootstrap are asymptotic results, we can not achieve the equicontinuity for every $n\in\mathbb{N}$, but only asymptotically.

As the estimators can be represented by a statistical operator which depends on the empirical measure it is crucial to concern stochastic processes which at last provide convergence of their empirical measure. Therefore, Strohriegl and Hable (2016) proposed to choose Varadarajan process. Let $(\Omega,\mathcal{A},\mu)$ be a probability space. Let $(Z_{i})_{i\in\mathbb{N}},$ $Z_{i}:\Omega\rightarrow\mathcal{Z}$, $i\in\mathbb{N}$, be a stochastic process and $\mathbf{W}_{n}:=(Z_{1},\ldots,Z_{n})$. Then the stochastic process $(Z_{i})_{i\in\mathbb{N}}$ is called a (strong) Varadarajan process if there exists a probability measure $P\in\mathcal{M}(\mathcal{Z})$ such that

[TABLE]

The stochastic process $(Z_{i})_{i\in\mathbb{N}}$ is called weak Varadarajan process if

[TABLE]

Examples for Varadarajan processes are certain Markov Chains, some mixing processes, ergodic process and processes which satisfy a law of large numbers for events in the sense of Steinwart et al. (2009, Definition 2.1), see Strohriegl and Hable (2016) for details.

2.1 Qualitative robustness for independent not identically distributed processes

In this section we relax the i.i.d. assumption in view of the identical distribution. We assume the random variables $Z_{i}$, $i\in\mathbb{N}$, to be independent, but not necessarily identically distributed.

The result below generalizes Christmann et al. (2013, Theorem 3) and Christmann et al. (2011), as the assumptions on the stochastic process are weaker as well as those on the statistical operator. Compared to Theorem 3 in Cuevas and Romo (1993), which shows qualitative robustness of the sequence of bootstrap estimators with values in $\mathbb{R}$, we have to strengthen the assumptions on the sample space, but do not need the estimator to be uniformly continuous. But keep in mind, that the assumption $d_{\textup{BL}}(P_{n},Q_{n})<\delta$ implies $d_{\textup{BL}}(P,Q)<\delta$, which is used for the i.i.d. case, in Christmann et al. (2013) and Cuevas and Romo (1993).

Theorem 2.2

*Let the sequence of estimators $(S_{n})_{n\in\mathbb{N}}$ be represented by a statistical operator $S:\mathcal{M}(\mathcal{Z})\rightarrow H$ via (1) for a Polish space $H$ and let $(\mathcal{Z},d_{\mathcal{Z}})$ be a totally bounded metric space.

Let $P_{\mathbb{N}}=\otimes_{i\in\mathbb{N}}P^{i}$, $P^{i}\in\mathcal{M}(\mathcal{Z})$ be an infinite product measure such that the coordinate process $(Z_{i})_{i\in\mathbb{N}}$, $Z_{i}\colon\mathcal{Z}^{\mathbb{N}}\rightarrow z_{i}$, $i\in\mathbb{N}$, is a strong Varadarajan process with limiting distribution $P$. Moreover define $\mathcal{P}:=\left\{Q_{\mathbb{N}}\in\mathcal{M}(\mathcal{Z}^{\mathbb{N}});\;Q_{\mathbb{N}}=\otimes_{i\in\mathbb{N}}Q^{i},\;Q^{i}\in\mathcal{M}(\mathcal{Z})\right\}$. Let $S:\mathcal{M}(\mathcal{Z})\rightarrow H$ be continuous at $P$ with respect to $d_{\textup{BL}}$ and let the estimators $S_{n}:\mathcal{Z}^{n}\rightarrow H,\;n\in\mathbb{N}$, be continuous.

Then the sequence of bootstrap approximations $(\mathcal{L}_{P_{n}^{*}}(S_{n}))_{n\in\mathbb{N}}$, is qualitatively robust at $P_{\mathbb{N}}$ with respect to $\mathcal{P}$. *

Remark 2.3

*The required properties on the statistical operator $S$ and on the sequence of estimators $(S_{n})_{n\in\mathbb{N}}$ in Theorem 2.2 ensure the qualitative robustness of $(S_{n})_{n\in\mathbb{N}}$, as long as the assumptions on the underlying stochastic processes are fulfilled.

The proof shows that the bootstrap approximation of every sequence of estimators $(S_{n})_{n\in\mathbb{N}}$ which is qualitatively robust in the sense of the definitions in Bustos (1980) and Strohriegl and Hable (2016, Definition 1) is qualitatively robust in the sense of Theorem 2.2.*

Hence Hampel’s theorem for the i.i.d. case can be generalized to bootstrap approximations and to the case of not necessarily identically distributed random variables if qualitative robustness is based on the definition of $\Pi$-robustness.

Unfortunately, the assumption on the space $(\mathcal{Z},d_{\mathcal{Z}})$ to be totally bounded seems to be necessary. In the proof of Theorem 2.2 we use a result of Dudley et al. (1991) to show uniformity on the space of probability measures $\mathcal{M}(\mathcal{Z})$. This result needs the bounded Lipschitz functions to be a uniform Glivenko-Cantelli class, which is equivalent to $(\mathcal{Z},d_{\mathcal{Z}})$ being totally bounded, see Dudley et al. (1991, Proposition 12). In order to weaken the assumption on $(\mathcal{Z},d_{\mathcal{Z}})$, probably another way to show uniformity on the space of probability measures $\mathcal{M}(\mathcal{Z})$ has to be found.

A short look on the metrics used on $\mathcal{Z}^{n}$ is advisable. We consider $\mathcal{Z}^{n}$ as the $n$-fold product space of the Polish space $(\mathcal{Z},d_{\mathcal{Z}})$. The product space $\mathcal{Z}^{n}$ is again a Polish space (in the product topology) and it is tempting to use a $p$-product metric $d_{n,p}$ on $\mathcal{Z}^{n}$, that is,

[TABLE]

where $\|\cdot\|_{p}$ is a $p_{n}$-norm on $\mathbb{R}^{n}$ for $1\leq p\leq\infty$. For example, $d_{n,2}$ is the Euclidean metric on $\mathbb{R}^{n}$ and d_{n,\infty}\big{(}(z_{1},\dots,z_{n}),(z_{1}^{\prime},\dots,z_{n}^{\prime})\big{)}=\max_{i}d(z_{i},z_{i}^{\prime}); all these metrics are strongly equivalent. However, these common metrics do not cover the intuitive meaning of qualitative robustness as the distance between two points in $\mathcal{Z}^{n}$ (i.e., two data sets) is small only if all coordinates are close together (small rounding errors). So points where only a small fraction of the coordinates are far-off (gross errors) are excluded. Using these metrics, the qualitative robustness of the sample mean at every $P_{\mathbb{N}}\in\mathcal{M}(\mathcal{Z}^{\mathbb{N}})$ can be shown, see e.g. Strohriegl and Hable (2016, Proposition 1). But the sample mean is a highly non-robust estimator, as gross errors have great impact on the estimate. Following Boente et al. (1987), we use the metric $d_{n}$ on $\mathcal{Z}^{n}:$

[TABLE]

This metric on $\mathcal{Z}^{n}$ covers both kinds of "small errors". Though $d_{n}$ is not strongly equivalent to $d_{n,p}$ in general, it is topologically equivalent to the $p$-product metrics $d_{n,p}$, see Strohriegl and Hable (2016, Lemma 1). Hence, $\mathcal{Z}^{n}$ is metrizable also with metric $d_{n}$. Moreover the continuity of $S_{n}$ on $\mathcal{Z}^{n}$ is with respect to the product topology on $\mathcal{Z}^{n}$ which can, due to the topological equivalence of these two metrics, be seen with respect to the common metrics $d_{n,p}$.

The next part gives two examples of stochastic processes of independent, but not necessarily identically distributed random variables, which are Varadarajan processes. In particular these stochastic processes even satisfy a strong law of large numbers for events (SLLNE) in the sense of Steinwart et al. (2009) and therefore are, due to Strohriegl and Hable (2016, Theorem 2), strong Varadarajan processes. The first example is rather simple and describes a sequence of univariate normal distributions.

Example* 1 Let $(a_{i})_{i\in\mathbb{N}}\subset\mathbb{R}$ be a sequence with $\lim_{i\rightarrow\infty}a_{i}=a\in\mathbb{R}$ and let $|a_{i}|\leq c$, for some constant $c>0$ for all $i\in\mathbb{N}$. Let $(Z_{i})_{i\in\mathbb{N}},$ $Z_{i}:\Omega\rightarrow\mathbb{R}$, be a stochastic process where $Z_{i}$, $i\in\mathbb{N}$, are independent and $Z_{i}\sim N(a_{i},1),\;i\in\mathbb{N}$. Then the process $(Z_{i})_{i\in\mathbb{N}}$ is a strong Varadarajan process.*

The second example are stochastic processes where the distributions of the random variables $Z_{i}$, $i\in\mathbb{N}$, are lying in a so-called shrinking $\varepsilon$-neighbourhood of a probability measure $P$.

Example* 2 Let $(\mathcal{Z},\mathcal{B})$ be a measurable space and let $(Z_{i})_{i\in\mathbb{N}}$ be a stochastic process with independent random variables $Z_{i}:\Omega\rightarrow\mathcal{Z}$, $Z_{i}\sim P^{i}$, where*

[TABLE]

for a sequence $\varepsilon_{i}\rightarrow 0$, $i\rightarrow\infty$, $\varepsilon_{i}>0$ and $\tilde{P}^{i},\;P\in\mathcal{M}(\mathcal{Z})$, $i\in\mathbb{N}$. Then the process $(Z_{i})_{i\in\mathbb{N}}$ is a strong Varadarajan process.

The next corollary shows, that Support Vector Machines are qualitatively robust. For a detailed introduction to Support Vector Machines see e.g., Schölkopf and Smola (2002) and Steinwart and Christmann (2008). Let $D_{n}:=(z_{1},z_{2},\ldots,z_{n})=((x_{1},y_{1}),(x_{2},y_{2}),\ldots,(x_{n},y_{n}))$ be a given dataset.

Corollary 2.4

*Let $\mathcal{Z}=\mathcal{X}\times\mathcal{Y}$, $\mathcal{Y}\subset\mathbb{R}$ closed, be a totally bounded, metric space and let $(Z_{i})_{i\in\mathbb{N}}$ be a stochastic process where the random variables $Z_{i}$, $i\in\mathbb{N}$, are independent and $Z_{i}\sim P^{i}:=(1-\varepsilon_{i})P+\varepsilon_{i}\tilde{P}^{i}$, $P,\tilde{P}^{i}\in\mathcal{M}(\mathcal{Z})$. Moreover let $(\lambda_{n})_{n\in\mathbb{N}}$ be a sequence of positive real valued numbers with $\lambda_{n}\rightarrow\lambda_{0},\;n\rightarrow\infty$, for some $\lambda_{0}>0$. Let $H$ be a reproducing kernel Hilbert space with continuous and bounded kernel $k$ and let $S_{\lambda_{n}}:(\mathcal{X}\times\mathcal{Y})^{n}\rightarrow H$ be the SVM estimator, which maps $D_{n}$ to $f_{L^{*},D_{n},\lambda_{n}}$ for a continuous and convex loss function $L:\mathcal{X}\times\mathcal{Y}\times\mathcal{Y}\rightarrow[0,\infty[$. It is assumed that $L(x,y,y)=0$ for every $(x,y)\in\mathcal{X}\times\mathcal{Y}$ and that $L$ is additionally Lipschitz continuous in the last argument.

Then we have for every $\varepsilon>0$ there is $\delta>0$ such that there is $n_{0}\in\mathbb{N}$ such that for all $n\geq n_{0}$ and for every process $(\tilde{Z}_{i})_{i\in\mathbb{N}}$, where $\tilde{Z}_{i}$ are independent and have distribution $Q^{i}$, $i\in\mathbb{N}$:*

[TABLE]

That is, the sequence of bootstrap approximations is qualitatively robust if the second (contaminated) process $(\tilde{Z}_{i})_{i\in\mathbb{N}}$ is still of the same kind, i.e. still independent, as the original uncontaminated process $(Z_{i})_{i\in\mathbb{N}}$.

2.2 Qualitative robustness for the moving block bootstrap of $\alpha$-mixing processes

Dropping the independence assumption we now focus on real valued mixing processes, in particular on strongly stationary $\alpha$-mixing or strong mixing stochastic processes. The mixing notion is an often used and well-accepted dependence notion which quantifies the degree of dependence of a stochastic process. There exist several types of mixing coefficients, but all of them are based on differences between probabilities $\mu(A_{1}\cap A_{2})-\mu(A_{1})\mu(A_{2})$. There is a large literature on this dependence structure. For a detailed overview on mixing, see Bradley (2005), Bradley (2007a, b, c), and Doukhan (1994) and the references therein. The $\alpha$-mixing structure has been introduced in Rosenblatt (1956). Also examples of relations between dependence structures and mixing coefficients can be found in the references above. Let $\Omega$ be a set equipped with two $\sigma$-algebras $\mathcal{A}_{1}$ and $\mathcal{A}_{2}$ and a probability measure $\mu$. Then the $\alpha$-mixing coefficient is defined by

[TABLE]

By definition the coefficients equal zero, if the $\sigma$-algebras are independent.

Moreover mixing can be defined for stochastic processes. We follow Steinwart et al. (2009, Definition 3.1):

Definition 2.5

Let $(Z_{i})_{i\in\mathbb{N}}$ be a stochastic process, $Z_{i}:\Omega\rightarrow\mathcal{Z}$, $i\in\mathbb{N}$, and let $\sigma(Z_{i})$ be the $\sigma$-algebra generated by $Z_{i}$, $i\in\mathbb{N}$. Then the $\alpha$-$bi$- and the $\alpha$-mixing coefficients are defined by

[TABLE]

A stochastic process $(Z_{i})_{i\in\mathbb{N}}$ is called $\alpha$- mixing with respect to $\mu$ if

[TABLE]

It is called weakly $\alpha$-$bi$-mixing with respect to $\mu$ if

[TABLE]

Instead of Efron’s empirical bootstrap another bootstrap approach is used in order to represent the dependence structure of an $\alpha$-mixing process. Künsch (1989) and Liu and Singh (1992) introduced the moving block bootstrap (MBB). Often resampling of single observations can not preserve the dependence structure of the process, therefore they decided to take blocks of length $b$ of observations instead. The dependence structure of the process is preserved, within these blocks. The block length $b$ increases with the number of observations $n$ for asymptotic considerations. A slight modification of the original moving block bootstrap, see for example Politis and Romano (1990) and Shao and Yu (1993), is used in the next two theorems in order to avoid edge effects.

The proofs are based on central limit theorems for empirical processes. There are several results concerning the moving block bootstrap of the empirical process in case of mixing processes, see for example Bühlmann (1994), Naik-Nimbalkar and Rajarshi (1994), and Peligrad (1998, Theorem 2.2) for $\alpha$-mixing sequences and Radulović (1996) and Bühlmann (1995) for $\beta$-mixing sequences. To our best knowledge there are so far no results concerning qualitative robustness for bootstrap approximations of estimators for $\alpha$-mixing stochastic processes. Therefore, Theorem 2.6 shows qualitative robustness for a stochastic process with values in $\mathbb{R}$. The proof is based on Peligrad (1998, Theorem 2.2), which provides a central limit theorem under assumptions on the process, which are weaker than those in Bühlmann (1994) and Naik-Nimbalkar and Rajarshi (1994). In the case of $\mathbb{R}^{d}$-valued, $d>1$, stochastic processes, stronger assumptions on the stochastic process are needed, as the central limit theorem in Bühlmann (1994) requires stronger assumptions, see Theorem 2.7.

Let $Z_{1},\ldots,Z_{n}$, $n\in\mathbb{N}$, be the first $n$ projections of a real valued stochastic process $(Z_{i})_{i\in\mathbb{N}}$ and let $b\in\mathbb{N},b<n$, be the block length. Then, for fixed $n\in\mathbb{N}$, the sample can be divided into blocks $B_{i,b}:=(Z_{i},\ldots,Z_{i+b-1})$. If $i>n-b+1$, we define $Z_{n+j}=Z_{j}$, for the missing elements of the blocks. To get the MBB bootstrap sample $\mathbf{W}^{*}_{n}=(Z^{*}_{1},\ldots,Z^{*}_{n})$, $\ell$ numbers $I_{1},\ldots,I_{\ell}$ from the set $\{1,\ldots,n\}$ are randomly chosen with replacement. Without loss of generality it is assumed that $n=\ell b$, if $n$ is not a multiple of $b$ we simply cut the last block, which is usually done in literature. Then the sample consists of the blocks $B_{I_{1},b},B_{I_{2},b},\ldots,B_{I_{\ell},b}$, that is $Z^{*}_{1}=Z_{I_{1}},Z^{*}_{2}=Z_{I_{1}+1},\ldots,Z^{*}_{b}=_{I_{1}+b-1},Z^{*}_{b+1}=Z_{I_{2}},\ldots,Z^{*}_{\ell b}=Z_{I_{\ell}+b-1}$.

As we are interested in estimators $S_{n}$, $n\in\mathbb{N}$, which can be represented by a statistical operator $S\colon\mathcal{M}(\mathcal{Z})\rightarrow H$ via $S(\mathbb{P}_{\mathbf{w}_{n}})=S_{n}(z_{1},\ldots,z_{n})$, for a Polish space $H$, see (1), the empirical measure of the bootstrap sample $\mathbb{P}_{\mathbf{W}^{*}_{n}}=\frac{1}{n}\sum_{i=1}^{n}\delta_{Z^{*}_{i}}$ should approximate the empirical measure of the original sample $\mathbb{P}_{\mathbf{W}_{n}}=\frac{1}{n}\sum_{i=1}^{n}\delta_{Z_{i}}$. Contrarily to qualitative robustness in the case of independent and not necessarily identically distributed random variables (Theorem 2.2), the assumptions on the statistical operator $S$ are strengthened for the case of $\alpha$-mixing sequences. In particular the statistical operator $S$ is assumed to be uniformly continuous for all $P\in(\mathcal{M}(\mathcal{Z}),d_{\textup{BL}})$. For the first theorem we assume the random variables $Z_{i},\;i\in\mathbb{N}$, to be real valued and bounded. Without loss of generality we assume $0\leq Z_{i}\leq 1$, otherwise a transformation leads to this assumption. For the bootstrap for the true as well as for the contaminated process, we assume the block length $b(n)$ and the number of blocks $\ell(n)$ to be sequences of integers satisfying

[TABLE]

$b(n)=b(2^{q})$ for $2^{q}\leq n<2^{q+1},\;q\in\mathbb{N},\;b(n)\rightarrow\infty$, $n\rightarrow\infty$ and $b(n)\cdot\ell(n)=n$, $n\in\mathbb{N}$.

Theorem 2.6

Let $P_{\mathbb{N}}\in\mathcal{M}(\mathbb{R}^{\mathbb{N}})$ be a probability measure on $(\mathbb{R}^{\mathbb{N}},\mathcal{B}^{\otimes\mathbb{N}})$ such that the coordinate process $\left(Z_{i}\right)_{i\in\mathbb{N}}$, $Z_{i}:\mathbb{R}^{\mathbb{N}}\rightarrow\mathbb{R}$ is bounded, strongly stationary, and $\alpha$-mixing with

[TABLE]

*Let $\mathcal{P}\subset\mathcal{M}(\mathbb{R}^{\mathbb{N}})$ be the set of probability measures such that the coordinate process fulfils the properties above for the same $\gamma>0$. Let $H$ be a Polish space, with some metric $d_{H}$, let $(S_{n})_{n\in\mathbb{N}}$ be a sequence of estimators which can be represented by a statistical operator $S:\mathcal{M}(\mathbb{R})\rightarrow H$ via (1). Moreover let $S_{n}$ be continuous and let $S$ be additionally uniformly continuous with respect to $d_{\textup{BL}}$. Then the sequence of estimators $(S_{n})_{n\in\mathbb{N}}$ is qualitatively robust at $P_{\mathbb{N}}$ with respect to $\mathcal{P}$. *

The assumptions on the stochastic process are on the one hand, together with the assumptions on the block length, used to ensure the validity of the bootstrap approximation and on the other hand, together with the assumptions on the statistical operator, respectively the sequence of estimators, to ensure the qualitative robustness.

The next theorem generalizes this result to stochastic processes with values in $[0,1]^{d}$, $d>1$, instead of $[0,1]\subset\mathbb{R}$. Therefore, for example, the bootstrap version of the SVM estimator is qualitatively robust under weak conditions. The proof of the next theorem follows the same lines as the proof of the theorem above, but another central limit theorem, which is shown in Bühlmann (1994), is used. Therefore the assumptions on the mixing property of the stochastic process are stronger and the random variables $Z_{i},\;i\in\mathbb{N},$ are assumed to have continuous marginal distributions. Again the bootstrap sample results of a moving block bootstrap where $\ell(n)$ blocks of length $b(n)$ are chosen, again assuming $\ell(n)\cdot b(n)=n$. Moreover, let $b(n)$ be a sequences of integers satisfying

[TABLE]

Theorem 2.7

Assume $\mathcal{Z}=[0,1]^{d}$, $d>1$. Let $P_{\mathbb{N}}$ be a probability measure such that the coordinate process $\left(Z_{i}\right)_{i\in\mathbb{N}}$, $Z_{i}:\mathcal{Z}^{\mathbb{N}}\rightarrow\mathcal{Z}$ is strongly stationary and $\alpha$-mixing with

[TABLE]

*Assume that $Z_{i}$ has continuous marginal distributions for all $i\in\mathbb{N}$. Define the set of probability measures $\mathcal{P}\subset\mathcal{M}(\mathcal{Z})$ such that the coordinate process is strongly stationary and $\alpha$-mixing as in (6).

Let $H$ be a Polish space, wit some metric $d_{H}$, $(S_{n})_{n\in\mathbb{N}}$ be a sequence of estimators such that $S_{n}:\mathcal{Z}^{n}\rightarrow H$ is continuous and assume that $S_{n}$ can be represented by a statistical operator $S:\mathcal{M}(\mathcal{Z})\rightarrow H$ via (1) which is additionally uniformly continuous with respect to $d_{\textup{BL}}$.*

*Then the sequence of estimators $(S_{n})_{n\in\mathbb{N}}$ is qualitatively robust at $P_{\mathbb{N}}$ with respect to $\mathcal{P}$. *

Although the assumptions on the statistical operator $S$, compared to Theorem 2.2, were strengthened in order to generalize the qualitative robustness to $\alpha$-mixing sequences in Theorem 2.6 and 2.7, M-estimators are still an example for qualitative robust estimators if the sample space $(\mathcal{Z},d_{\mathcal{Z}})$, $\mathcal{Z}\subset\mathbb{R}$ is compact. The compactness of $(\mathcal{Z},d_{\mathcal{Z}})$ implies the compactness of the space $(\mathcal{M}(\mathcal{Z}),d_{\textup{BL}})$, see Parthasarathy (1967, Theorem 6.4). As the statistical operator $S$ is continuous, the compactness of $\mathcal{M}(\mathcal{Z})$ implies the uniform continuity of $S$. Another example of M-estimators which are uniformly continuous even if the input space is not compact is given in Cuevas and Romo (1993, Theorem 4).

Acknowledgements: This research was partially supported by the DFG Grant 291/2-1 "Support Vector Machines bei stochastischer Unabhängigkeit". Moreover I would like to thank Andreas Christmann for helpful discussions on this topic.

3 Proofs

This section contains the proofs of the main theorems and corollaries.

3.1 Proofs of Section 2.1

Before proving Theorem 2.1, we state a rather technical lemma, connecting the product measure $\otimes_{i=1}^{n}P^{i}\in\mathcal{M}(\mathcal{Z}^{n})$ of independent random variables to their mixture measure $\frac{1}{n}\sum_{i=1}^{n}P^{i}\in\mathcal{M}(\mathcal{Z})$. Let $(\mathcal{Z},d_{\mathcal{Z}})$ be a Polish space.

Lemma 3.1

Let $P_{n},Q_{n}\in\mathcal{M}(\mathcal{Z}^{n})$ such that $P_{n}=\otimes_{i=1}^{n}P^{i}$ and $Q_{n}=\otimes_{i=1}^{n}Q^{i}$, $P^{i},Q^{i}\in\mathcal{M}(\mathcal{Z}),\;i\in\mathbb{N}$. Then for all $\delta>0$:

[TABLE]

Proof: Let $\textup{BL}_{1}$ be the set of bounded Lipschitz functions with $\|f\|_{\textup{BL}}\leq 1$.By assumption we have $d_{\textup{BL}}(P_{n},Q_{n})\leq\delta$. Moreover for a function $f:\mathcal{Z}\rightarrow\mathbb{R}$:

[TABLE]

Then,

[TABLE]

Now every function $f\in\textup{BL}_{1}(\mathcal{Z})$ can be identified as a function $\tilde{f}:\mathcal{Z}^{n}\rightarrow\mathcal{Z}$, $(z_{1},\ldots,z_{n})\mapsto\tilde{f}(z_{1},\ldots,z_{n}):=f(z_{i})$. This function is also Lipschitz continuous on $\mathcal{Z}^{n}:$

[TABLE]

where $d_{\mathcal{Z}}(z_{1},z^{\prime}_{1})+\ldots+d_{\mathcal{Z}}(z_{i},z^{\prime}_{i})+\ldots+d_{\mathcal{Z}}(z_{n},z^{\prime}_{n})$ induces the product topology on $\mathcal{Z}^{n}$. That is $\tilde{f}\in\textup{BL}_{1}(\mathcal{Z}^{n})$. Note that this is also true for every $p$-product metric $d_{n,p}$ in $\mathcal{Z}^{n}$, $1\leq p\leq\infty$, as they are strongly equivalent. Hence,

[TABLE]

which yields the assertion. $\square$

Proof of Theorem 2.2: To prove Theorem 2.2 we first use the triangle inequality to split the bounded Lipschitz distance between the distribution of the estimator $S_{n}$, $n\in\mathbb{N}$, into two parts regarding the distribution of the estimator under the joint distribution $P_{n}$ of $(Z_{1},\ldots,Z_{n})$:

[TABLE]

Then the representation of the estimator $S_{n}$ by the statistical operator $S$ and the continuity of this operator in $P$ together with the Varadarajan property and the independence assumption on the stochastic process yield the assertion.

First we regard part I: Define the distribution $P_{\mathbb{N}}\in\mathcal{M}(\mathcal{Z}^{\mathbb{N}})$ and let $P^{*}_{\mathbb{N}}$ be the bootstrap approximation of $P_{\mathbb{N}}$. Define, for $n\in\mathbb{N}$, the random variables

$\mathbf{W}_{n}:\mathcal{Z}^{\mathbb{N}}\rightarrow\mathcal{Z}^{n},\;\mathbf{W}_{n}=(Z_{1},\ldots,Z_{n})$, $z_{\mathbb{N}}\mapsto\mathbf{W}_{n}(z_{\mathbb{N}})=\mathbf{w}_{n}=(z_{1},\ldots,z_{n})$, and

$\mathbf{W}^{\prime}_{n}:\mathcal{Z}^{\mathbb{N}}\rightarrow\mathcal{Z}^{n},\;\mathbf{W}^{\prime}_{n}=(Z^{\prime}_{1},\ldots,Z^{\prime}_{n})$, $z_{\mathbb{N}}\mapsto\mathbf{w}^{\prime}_{n}$,

such that $\mathbf{W}_{n}(P_{\mathbb{N}})=P_{n}$ and $\mathbf{W}^{\prime}_{n}(P_{\mathbb{N}}^{*})=P^{*}_{n}$.

Denote the bootstrap sample by $\mathbf{W}^{*}_{n}:=(Z^{*}_{1},\ldots,Z^{*}_{n})$, $\mathbf{W}^{*}_{n}:\mathcal{Z}^{\mathbb{N}}\rightarrow\mathcal{Z}^{n}$, $z_{\mathbb{N}}\mapsto\mathbf{w}^{*}_{n}$.

As Efron’s empirical bootstrap is used, the bootstrap sample, which is chosen via resampling with replacement out of $Z_{1},\ldots,Z_{\ell}$, $\ell\in\mathbb{N}$, has distribution $Z^{*}_{i}\sim\mathbb{P}_{\mathbf{W}_{\ell}}=\frac{1}{\ell}\sum_{j=1}^{\ell}\delta_{Z_{j}}$, $i\in\mathbb{N}$, respectively $\mathbf{W}^{*}_{n}:=(Z^{*}_{1},\ldots,Z^{*}_{n})\sim\otimes_{i=1}^{n}\mathbb{P}_{\mathbf{W}_{\ell}}$. The bootstrap approximation of $P_{\ell}$, $\ell\in\mathbb{N}$, is the empirical measure of the bootstrap sample $P^{*}_{\ell}=\otimes_{i=1}^{\ell}\frac{1}{n}\sum_{j=1}^{n}\delta_{Z^{*}_{j}}$.

Further denote the joint distribution of $\mathbf{W}_{\mathbb{N}}$, $\mathbf{W}^{*}_{\mathbb{N}}$, and $\mathbf{W}^{\prime}_{\mathbb{N}}$ by $K_{\mathbb{N}}\in\mathcal{M}(\mathcal{Z}^{\mathbb{N}}\times\mathcal{Z}^{\mathbb{N}}\times\mathcal{Z}^{\mathbb{N}})$. Then, $K_{\mathbb{N}}$ has marginal distributions $K_{\mathbb{N}}(B_{1}\times\mathcal{Z}^{\mathbb{N}}\times\mathcal{Z}^{\mathbb{N}})=P_{\mathbb{N}}(B_{1})$ for all $B_{1}\in\mathcal{B}^{\otimes\mathbb{N}}$, $K_{\mathbb{N}}(\mathcal{Z}^{\mathbb{N}}\times B_{2}\times\mathcal{Z}^{\mathbb{N}})=\otimes_{i\in\mathbb{N}}\mathbb{P}_{\mathbf{W}_{n}}(B_{2})$ for all $B_{2}\in\mathcal{B}^{\otimes\mathbb{N}}$, and $K_{\mathbb{N}}(\mathcal{Z}^{\mathbb{N}}\times\mathcal{Z}^{\mathbb{N}}\times B_{3})=P_{\mathbb{N}}^{*}(B_{3})$ for all $B_{3}\in\mathcal{B}^{\otimes\mathbb{N}}$.

Then,

[TABLE]

and therefore

[TABLE]

By assumption the coordinate process $(Z_{i})_{i\in\mathbb{N}}$ consists of independent random variables, hence we have $P_{n}=\otimes_{i=1}^{n}P^{i}$, for $P^{i}=Z_{i}(P_{\mathbb{N}})$, $i\in\mathbb{N}$.

Moreover $(\mathcal{Z},d_{\mathcal{Z}})$ is assumed to be a totally bounded metric space. Then, due to Dudley et al. (1991, Proposition 12), the set $\textup{BL}_{1}(\mathcal{Z},d_{\mathcal{Z}})$ is a uniform Glivenko-Cantelli class. That is, if $Z_{i}\sim P$ i.i.d. $i\in\mathbb{N}$, we have for all $\eta>0$:

[TABLE]

Applying this to the bootstrap sample $(Z^{*}_{1},\ldots,Z^{*}_{m})$, $m\in\mathbb{N}$, which is found by resampling with replacement out of the original sample $(Z_{1},\ldots,Z_{n})$, we have, for all $\mathbf{w}_{n}\in\mathcal{Z}^{n}$,

[TABLE]

Let $\varepsilon>0$ be arbitrary but fixed. Then, for every $\delta_{0}>0$ there is $n_{1}\in\mathbb{N}$ such that for all $n\geq n_{1}$ and all $\mathbb{P}_{\mathbf{w}_{n}}\in\mathcal{M}(\mathcal{Z})$:

[TABLE]

And, using the same argumentation for the sequence of random variables $Z^{\prime}_{i}$, $i\in\mathbb{N}$, which are i.i.d. and have distribution $\frac{1}{n}\sum_{i=1}^{n}\delta_{Z^{*}_{i}}=\mathbb{P}_{\mathbf{W}^{*}_{n}}$:

[TABLE]

Respectively, for every $\delta_{0}>0$ there is $n_{2}\in\mathbb{N}$ such that for all $n\geq n_{2}$ and all $\mathbb{P}_{\mathbf{w}^{*}_{n}}\in\mathcal{M}(\mathcal{Z})$:

[TABLE]

As the process $(Z_{i})_{i\in\mathbb{N}}$ is a strong Varadarajan process by assumption, there exists a probability measure $P\in\mathcal{M}(\mathcal{Z})$ such that

[TABLE]

That is, for every $\delta_{0}>0$ there is $n_{3}\in\mathbb{N}$ such that for all $n\geq n_{3}$:

[TABLE]

The continuity of the statistical operator $S:\mathcal{M}(\mathcal{Z})\rightarrow H$ in $P\in\mathcal{M}(\mathcal{Z})$ yields: for every $\varepsilon>0$ there exists $\delta_{0}>0$ such that for all $Q\in\mathcal{M}(\mathcal{Z})$:

[TABLE]

As the Prohorov metric $\pi_{d_{H}}$ is bounded by the Ky Fan metric, see Dudley (1989, Theorem 11.3.5) we conclude:

[TABLE]

Due to the definition of the statistical operator $S$, this is equivalent to

[TABLE]

The triangle inequality

[TABLE]

and the continuity of the statistical operator $S$, see (11), then yield, for all $\varepsilon>0$,

[TABLE]

Using the triangle inequality,

[TABLE]

gives for all $n\geq\max\{n_{1},n_{2},n_{3}\}$:

[TABLE]

Hence, for all $\varepsilon>0$ there are $n_{1},n_{2},n_{3}\in\mathbb{N}$ such that vor all $n\geq\max\{n_{1},n_{2},n_{3}\}$, the infimum in (12) is bounded by $\frac{\varepsilon}{2}$. Therefore

[TABLE]

The equivalence between the Prohorov metric and the bounded Lipschitz metric for Polish spaces, see Huber (1981, Chapter 2, Corollary 4.3), yields the existence of $n_{0,1}\in\mathbb{N}$ such that for all $n\geq n_{0,1}:$

[TABLE]

To prove the convergence of the term in part II, consider the distribution $Q_{\mathbb{N}}\in\mathcal{M}(\mathcal{Z}^{\mathbb{N}})$ and let $Q^{*}_{\mathbb{N}}$ be the bootstrap approximation of $Q_{\mathbb{N}}$. Define, for $n\in\mathbb{N}$, the random variables

$\tilde{\mathbf{W}}_{n}:\mathcal{Z}^{\mathbb{N}}\rightarrow\mathcal{Z}^{n},\;\tilde{\mathbf{W}}_{n}=(\tilde{Z}_{1},\ldots,\tilde{Z}_{n})$, $z_{\mathbb{N}}\mapsto\tilde{\mathbf{w}}_{n}$ with distribution $\tilde{\mathbf{W}}_{n}(Q_{\mathbb{N}})=Q_{n}$,

$\tilde{\mathbf{W}}^{\prime}_{n}:\mathcal{Z}^{\mathbb{N}}\rightarrow\mathcal{Z}^{n},\;\tilde{\mathbf{W}}^{\prime}_{n}=(\tilde{Z}^{\prime}_{1},\ldots,\tilde{Z}^{\prime}_{n})$, $z_{\mathbb{N}}\mapsto\tilde{\mathbf{w}}^{\prime}_{n}$, with distribution $\tilde{\mathbf{W}}^{\prime}_{n}(Q^{*}_{\mathbb{N}})=Q^{*}_{n}$, and

the bootstrap sample $\tilde{\mathbf{W}}^{*}_{n}:\mathcal{Z}^{\mathbb{N}}\rightarrow\mathcal{Z}^{n},\;\tilde{\mathbf{W}}^{*}_{n}=(\tilde{Z}^{*}_{1},\ldots,\tilde{Z}^{*}_{n})$, $z_{\mathbb{N}}\mapsto\tilde{\mathbf{w}}^{*}_{n}$, with distribution $\otimes_{i=1}^{n}\mathbb{Q}_{\tilde{\mathbf{W}}_{\ell}}=\otimes_{i=1}^{n}\frac{1}{\ell}\sum_{i=1}^{\ell}\delta_{\tilde{Z}_{i}}.$

Moreover let $\tilde{K}_{\mathbb{N}}\in\mathcal{M}(\mathcal{Z}^{\mathbb{N}}\times\mathcal{Z}^{\mathbb{N}}\times\mathcal{Z}^{\mathbb{N}}\times\mathcal{Z}^{\mathbb{N}})$ denote the joint distribution of $\mathbf{W}_{\mathbb{N}}$, $\tilde{\mathbf{W}}_{\mathbb{N}}$, $\tilde{\mathbf{W}}^{*}_{\mathbb{N}}$, and $\tilde{\mathbf{W}}^{\prime}_{\mathbb{N}}$. Then, $\tilde{K}_{\mathbb{N}}\in\mathcal{M}(\mathcal{Z}^{\mathbb{N}}\times\mathcal{Z}^{\mathbb{N}}\times\mathcal{Z}^{\mathbb{N}}\times\mathcal{Z}^{\mathbb{N}})$ has marginal distributions $P_{\mathbb{N}}$, $Q_{\mathbb{N}}$, $\otimes_{i\in\mathbb{N}}\mathbb{Q}_{\tilde{\mathbf{W}}_{n}}$, and $Q_{\mathbb{N}}^{*}$.

First, similar to the argumentation for part I, Efron’s bootstrap and Dudley et al. (1991, Proposition 12) give for $\tilde{\mathbf{w}}_{n}\in\mathcal{Z}^{n}$:

[TABLE]

Hence, for arbitrary, but fixed $\varepsilon>0$, for every $\delta_{0}>0$ there is $n_{4}\in\mathbb{N}$ such that for all $n\geq n_{4}$ and all $\mathbb{Q}_{\tilde{\mathbf{w}}_{n}}\in\mathcal{M}(\mathcal{Z})$:

[TABLE]

Further,

[TABLE]

Respectively, for every $\delta_{0}>0$ there is $n_{5}\in\mathbb{N}$ such that for all $n\geq n_{5}$ and all $\mathbb{Q}_{\tilde{\mathbf{w}}^{*}_{n}}=\frac{1}{n}\sum_{i=1}^{n}\delta_{\tilde{z}^{*}_{i}}\in\mathcal{M}(\mathcal{Z})$:

[TABLE]

Moreover, as the random variables $Z_{i}$, $Z_{i}\sim P^{i},$ $i\in\mathbb{N}$, are independent, the bounded Lipschitz distance between the empirical measure and $\frac{1}{n}\sum_{i=1}^{n}P^{i}$ can be bounded, due to Dudley et al. (1991, Theorem 7). As totally bounded spaces are particularly separable, see Denkowski et al. (2003, below Corollary 1.4.28), Dudley et al. (1991, Proposition 12) provides that $\textup{BL}_{1}(\mathcal{Z},d_{\mathcal{Z}})$ is a uniform Glivenko-Cantelli class. The proof of this proposition does not depend on the distributions of the random variables $Z_{i},\;i\in\mathbb{N}$, and is therefore also valid for independent and not necessarily identically distributed random variables. Hence Dudley et al. (1991, Theorem 7) yields for all $\eta>0$:

[TABLE]

as long as the assumptions of Proposition 12 in Dudley et al. (1991) apply. As $\textup{BL}_{1}(\mathcal{Z},d_{\mathcal{Z}})$ is bounded, we have $\mathcal{F}_{0}=\textup{BL}_{1}(\mathcal{Z},d_{\mathcal{Z}})$, see Dudley et al. (1991, page 499, before Proposition 10), hence it is sufficient to show that $\textup{BL}_{1}(\mathcal{Z},d_{\mathcal{Z}})$ is image admissible Suslin. By assumption $(\mathcal{Z},d_{\mathcal{Z}})$ is totally bounded, hence $\textup{BL}_{1}(\mathcal{Z},d_{\mathcal{Z}})$ is separable with respect to $\|\cdot\|_{\infty}$, see Strohriegl and Hable (2016, Lemma 3). As $f\in\textup{BL}_{1}(\mathcal{Z},d_{\mathcal{Z}})$ implies $\|f\|_{\infty}\leq 1$, the space $\textup{BL}_{1}(\mathcal{Z},d_{\mathcal{Z}})$ is a bounded subset of $(\mathit{C}_{b}(\mathcal{Z},d_{\mathcal{Z}}),\|\cdot\|_{\infty})$, which is due to Dudley (1989, Theorem 2.4.9) a complete space. Now, $\textup{BL}_{1}(\mathcal{Z},d_{\mathcal{Z}})$ is a closed subset of $(\mathit{C}_{b}(\mathcal{Z},d_{\mathcal{Z}}),\|\cdot\|_{\infty})$ with respect to $\|\cdot\|_{\infty}$. Hence $\textup{BL}_{1}(\mathcal{Z},d_{\mathcal{Z}})$ is complete, due to Denkowski et al. (2003, Proposition 1.4.17). Therefore $\textup{BL}_{1}(\mathcal{Z},d_{\mathcal{Z}})$ is separable and complete with respect to $\|\cdot\|_{\infty}$ and particularly a Suslin space, see Dudley (2014, p.229). As Lipschitz continuous functions are also equicontinuous, Dudley (2014, Theorem 5.28 (c)) gives that $\textup{BL}_{1}(\mathcal{Z},d_{\mathcal{Z}})$ is image admissible Suslin.

Hence, Dudley et al. (1991, Theorem 7) yields

[TABLE]

and

[TABLE]

That is, there is $n_{6}\in\mathbb{N}$ such that for all $n\geq n_{6}$

[TABLE]

Moreover, due to Lemma 3.1, we have

[TABLE]

Then the strong Varadarajan property of $(Z_{i})_{i\in\mathbb{N}}$ yields that there is $n_{7}\in\mathbb{N}$ such that for all $n\geq n_{7}:$

[TABLE]

Similar to the argumentation for part I we conclude, using again the boundedness of the Prohorov metric $\pi_{d_{H}}$ by the Ky Fan metric, see Dudley (1989, Theorem 11.3.5):

[TABLE]

Due to the definition of the statistical operator $S$, this is equivalent to

[TABLE]

Moreover the triangle inequality yields

[TABLE]

Hence, for all $n\geq\max\{n_{4},n_{5},n_{6},n_{7}\}$, we obtain

[TABLE]

The continuity of the statistical operator $S$ in $P$, see (11), gives

[TABLE]

Further, the triangle inequality yields

[TABLE]

Therefore we conclude, for all $n\geq\max\{n_{4},n_{5},n_{6},n_{7}\}$,

[TABLE]

[TABLE]

Now, assume $d_{\textup{BL}}(P_{n},Q_{n})\leq\frac{\delta_{0}}{6}$ , then (20) yields $d_{\textup{BL}}\left(\frac{1}{n}\sum_{i=1}^{n}P^{i},\frac{1}{n}\sum_{i=1}^{n}Q^{i}\right)\leq\frac{\delta_{0}}{6}$, therefore this term can be omitted. Note that this is only proven for the $p$-product metrics on $\mathcal{Z}^{n}$ and not for the metric $d_{n}$ from (4). For this metric we need a different argumentation, which is stated below the next calculation.

Hence, for all $n\geq\max\{n_{4},n_{5},n_{6},n_{7}\}$,

[TABLE]

In order to show the above bound for the metric $d_{n}$, see (4), on $\mathcal{Z}^{n}$, we use another variant of the triangle inequality in (22):

[TABLE]

Assume $d_{\textup{BL}}(P_{n},Q_{n})\leq\frac{\delta^{2}_{0}}{64}$. Then, the strong equivalence between the Prohorov metric and the bounded Lipschitz metric on Polish spaces, see Huber (1981, Chapter 2, Corollary 4.3), yields $\pi_{d_{n}}(P_{n},Q_{n})\leq\sqrt{d_{\textup{BL}}(P_{n},Q_{n})}\leq\frac{\delta_{0}}{8}$. Due to Dudley (1989, Theorem 11.6.2), $\pi_{d_{n}}(P_{n},Q_{n})\leq\frac{\delta_{0}}{8}$ implies the existence of a probability measure $\mu\in\mathcal{M}(\mathcal{Z}^{n}\times\mathcal{Z}^{n})$ with marginal distributions $P_{n}$ and $Q_{n}$, such that $\mu\left(\left\{(\mathbf{w}_{n},\tilde{\mathbf{w}}_{n})\in\mathcal{Z}^{n}\times\mathcal{Z}^{n}\;|\;d_{n}(\mathbf{w}_{n},\tilde{\mathbf{w}}_{n})>\frac{\delta_{0}}{8}\right\}\right)\leq\frac{\delta_{0}}{8}$. By a simple calculation $d_{n}(\mathbf{w}_{n},\tilde{\mathbf{w}}_{n})\leq\frac{\delta_{0}}{8}$ implies $\pi_{d_{n}}\left(\frac{1}{n}\sum_{i=1}^{n}\delta_{z_{i}},\frac{1}{n}\sum_{i=1}^{n}\delta_{\tilde{z}_{i}}\right)\leq\frac{\delta_{0}}{8}$ and we have:

[TABLE]

Again the equivalence between the metrics $\pi$ and $d_{\textup{BL}}$ yields:

[TABLE]

Now we choose the joint distribution $\tilde{K}_{\mathbb{N}}$ of $\mathbf{W}_{\mathbb{N}}$, $\tilde{\mathbf{W}}_{\mathbb{N}}$, $\tilde{\mathbf{W}}^{*}_{\mathbb{N}}$, and $\tilde{\mathbf{W}}^{\prime}_{\mathbb{N}}$ such that the distribution of $(\mathbf{W}_{n},\tilde{\mathbf{W}}_{n}):\mathcal{Z}^{\mathbb{N}}\times\mathcal{Z}^{\mathbb{N}}\rightarrow\mathcal{Z}^{n}\times\mathcal{Z}^{n}$ is $\mu\in\mathcal{M}(\mathcal{Z}^{n}\times\mathcal{Z}^{n})$. Then we conclude:

[TABLE]

Now, adapting the inequalities in (16), (17), and (21) in $\varepsilon$ respectively $n$ yields the boundedness of the above term by $\frac{\varepsilon}{2}$ for $d_{\textup{BL}}(P_{n},Q_{n})\leq\frac{\delta^{2}_{0}}{64}$ and for all $n\geq\{n_{4},n_{5},n_{7}\}$.

Now we can go on with the proof similar for both kinds of metrics on $\mathcal{Z}^{n}$.

The equivalence between the Prohorov metric and the bounded Lipschitz metric on Polish spaces, see Huber (1981, Chapter 2, Corollary 4.3), yields the existence of $n_{0,2}\in\mathbb{N}$ such that for all $n\geq n_{0,2}$, $d_{\textup{BL}}(P_{n},Q_{n})\leq\frac{\delta_{0}}{6}$ (respectively $d_{\textup{BL}}(P_{n},Q_{n})\leq\frac{\delta^{2}_{0}}{64}$) implies

[TABLE]

Now, (15) and (24) yield for all $n\geq\max\{n_{0,1},n_{0,2}\}$:

[TABLE]

Recall that $\mathcal{L}_{P_{n}^{*}}(S_{n})=:\zeta_{n}$ and $\mathcal{L}_{Q_{n}^{*}}(S_{n})=:\xi_{n}$ are random quantities with values in $\mathcal{M}(H)$. Hence (25) is equivalent to

[TABLE]

respectively

[TABLE]

Therefore, for all $f\in\textup{BL}_{1}(\mathcal{M}(\mathcal{Z}))$ and for all $n\geq\max\{n_{0,1},n_{0,2}\}$:

[TABLE]

by a variant of Strassen’s Theorem, see Huber (1981, Chapter 2, Theorem 4.2, (2)$\Rightarrow$(1)). That is,

[TABLE]

Hence for every $\varepsilon>0$ we find $\delta=\frac{\delta_{0}}{6}$ and $n_{0}=\max\{n_{0,1},n_{0,2}\}$ such that for all $n\geq n_{0}$:

[TABLE]

which yields the assertion. $\square$

Proof of Example 1:

Without any restriction we assume $a=0$. Otherwise regard the process $Z_{i}-a$, $i\in\mathbb{N}$. By assumption, the random variables $Z_{i}$, $i\in\mathbb{N}$, are independent. Hence $I_{B}\circ Z_{i}$, $i\in\mathbb{N}$, are independent, see for example Hoffmann-Jørgensen (1994, Theorem 2.10.6) for all measurable $B\in\mathcal{B}$, as $I_{B}$ is a measurable function. According to Steinwart et al. (2009, Proposition 2.8), $(Z_{i})_{i\in\mathbb{N}}$ satisfies the SLLNE if there is a probability measure $P$ in $\mathcal{M}(\mathcal{Z})$ such that $\lim_{n\rightarrow\infty}\frac{1}{n}\sum_{i=1}^{n}\mathbb{E}_{\mu}I_{B}\circ Z_{i}=P(B)$ for all measurable $B\in\mathcal{B}$. Hence:

[TABLE]

where $f_{i}(x)=\frac{1}{\sqrt{2\pi}}e^{-\frac{1}{2}(x-a_{i})^{2}}$ denotes the density of the normal distribution $N(0,1)$ with respect to the Lebesgue measure $\lambda^{1}$. Moreover define $g:\mathbb{R}\rightarrow\mathbb{R}$ by

[TABLE]

Therefore $|f_{i}|\leq|g|$, for all $i\in\mathbb{N}$, $g$ is integrable and due to Lebesgue’s Theorem, see for example Hoffmann-Jørgensen (1994, Theorem 3.6):

[TABLE]

We have $f_{i}\rightarrow f_{0}$, where $f_{0}=\frac{1}{\sqrt{2\pi}}e^{-\frac{1}{2}x^{2}}$ for all $x\in\mathbb{R}$, as $a_{i}\rightarrow 0$ and therefore the Lemma of Kronecker, see for example Hoffmann-Jørgensen (1994, Theorem 4.9, Equation 4.9.1) yields: $\lim_{n\rightarrow\infty}\frac{1}{n}\sum_{i=1}^{n}f_{i}(x)=f_{0}(x)$ for all $x\in\mathcal{X}$.

Now (26) yields the SLLNE:

[TABLE]

With Strohriegl and Hable (2016, Zheorem 2) the Varadarajan property is given. $\square$

Proof of Example 2:

Similar to the proof of Example 1, we first show the SLLNE, that is there exists a probability measure $P\in\mathcal{M}(\mathcal{Z})$ such that

[TABLE]

Now let $B\subset\Omega$ be an arbitrary measurable set. Then:

[TABLE]

As, $0\leq\frac{1}{n}\sum_{i=1}^{n}\varepsilon_{i}\int I_{B}\,dP\leq\frac{1}{n}\sum_{i=1}^{n}\varepsilon_{i}$ and $\varepsilon_{i}\rightarrow 0$, we have

[TABLE]

and similarly

[TABLE]

Hence (27) yields

[TABLE]

and therefore, due to Strohriegl and Hable (2016, Theorem 2), the assertion. $\square$

Proof of Corollary 2.4:

Due to Example 2, the stochastic process is a Varadarajan process. Hable and Christmann (2011, Theorem 3.2) ensures the continuity of the statistical operator $S:\mathcal{M}(\mathcal{Z})\rightarrow H,\;P\mapsto f_{L^{*},P,\lambda}$ for a fixed value $\lambda\in(0,\infty)$. Moreover Hable and Christmann (2011, Corollary 3.4) yields the continuity of the estimator $S_{n}:\mathcal{Z}^{n}\rightarrow H,\;D_{n}\mapsto f_{L^{*},D_{n},\lambda}$ for every fixed $\lambda\in(0,\infty)$. Hence for fixed $\lambda>0$ the bootstrap approximation of the SVM estimator is qualitatively robust, for the given assumptions. Moreover the proof of Theorem 2.2, equation (25), and the equivalence between between bounded Lipschitz metric and Prokhorov distance yield: for every $\varepsilon>0$ there is $\delta>0$ such that there is $n_{0}\in\mathbb{N}$ such that for all $n\geq n_{0}$ and if $d_{\textup{BL}}(P_{n},Q_{n})\leq\delta$:

[TABLE]

Similarly to the proof of the qualitative robustness in Strohriegl and Hable (2016, Theorem 4) we get: for every $\varepsilon>0$ there is $n_{\varepsilon}$, such that for all $n\geq n_{\varepsilon}$:

[TABLE]

And the same argumentation as in the proof of the qualitative robustness of the SVM estimator for the non-i.i.d. case in Strohriegl and Hable (2016, Theorem 4) for the cases $n_{0}\leq n\leq n_{\varepsilon}$ and $n>n_{\varepsilon}$ yields the assertion. $\square$

3.2 Proofs of Section 2.2

Proof of Theorem 2.6:

Proof of Theorem 2.6: Let $P^{*}_{\mathbb{N}},Q^{*}_{\mathbb{N}}\in\mathcal{M}(\mathcal{Z}^{\mathbb{N}})$ be the bootstrap approximations of the true distribution $P_{\mathbb{N}}$ and the contaminated distribution $Q_{\mathbb{N}}$. First, the triangle inequality yields:

[TABLE]

First, we regard the term in part II. Let $\sigma(Z_{i}),\;i\in\mathbb{N}$, be the $\sigma$-algebra generated by $Z_{i}$. Due to the assumptions on the mixing process $\sum_{m>n}\alpha(\sigma(Z_{1},\ldots,Z_{i}),\sigma(Z_{i+m},\ldots),P_{\mathbb{N}})=\mathcal{O}(n^{-\gamma}),\;i\in\mathbb{N},\gamma>0$, the sequence $\left(\alpha(\sigma(Z_{1},\ldots,Z_{i}),\sigma(Z_{i+m},\ldots),\mu)\right)_{m\in\mathbb{N}}$ is a null sequence. Moreover it is bounded by the definition of the $\alpha$-mixing coefficient which, due to the strong stationarity, does not depend on $i$. Therefore

[TABLE]

Hence, the process is weakly $\alpha$-$bi$-mixing with respect to $P_{\mathbb{N}}$, see Definition 2.5. Due to the stationarity assumption, the process $(Z_{i})_{i\in\mathbb{N}}$ is additionally asymptotically mean stationary, that is $\lim_{n\rightarrow\infty}\frac{1}{n}\sum_{i=1}^{n}\mathbb{E}I_{B}\circ Z_{i}=P(B)$ for all $B\in\mathcal{A}$ for a probability measure $P$. Therefore the process satisfies the WLLNE, see Steinwart et al. (2009, Proposition 3.2), and therefore is a weak Varadarajan process, see Strohriegl and Hable (2016, Theorem 2).

As the process is assumed to be a Varadarajan process and due to the assumptions on the sequence of estimators $(S_{n})_{n\in\mathbb{N}},$ qualitative robustness of $\left(S_{n}\right)_{n\in\mathbb{N}}$ is ensured by Strohriegl and Hable (2016, Theorem 1). Together with the equivalence between the Prohorov metric and the bounded Lipschitz metric for Polish spaces, see Huber (1981, Chapter 2, Corollary 4.3), it follows:

For every $\varepsilon>0$ there is $\delta>0$ such that for all $n\in\mathbb{N}$ and for all $Q_{n}\in\mathcal{M}(\mathcal{Z}^{n})$ we have:

[TABLE]

This implies

[TABLE]

Hence the convergence of the term in part II is shown.

To prove the convergence of the term in part I, consider the distribution $P_{\mathbb{N}}\in\mathcal{M}(\mathcal{Z}^{\mathbb{N}})$ and let $P^{*}_{\mathbb{N}}$ be the bootstrap approximation of $P_{\mathbb{N}}$, via the blockwise bootstrap. Define, for $n\in\mathbb{N}$, the random variables

$\mathbf{W}_{n}:\mathcal{Z}^{\mathbb{N}}\rightarrow\mathcal{Z}^{n},\;\mathbf{W}_{n}=(Z_{1},\ldots,Z_{n})$, $z_{\mathbb{N}}\mapsto\mathbf{w}_{n}=(z_{1},\ldots,z_{n})$, and

$\mathbf{W}^{\prime}_{n}:\mathcal{Z}^{\mathbb{N}}\rightarrow\mathcal{Z}^{n},\;\mathbf{W}^{\prime}_{n}=(Z^{\prime}_{1},\ldots,Z^{\prime}_{n})$, $z_{\mathbb{N}}\mapsto\mathbf{w}^{\prime}_{n}$,

such that $\mathbf{W}_{n}(P_{\mathbb{N}})=P_{n}$ and $\mathbf{W}^{\prime}_{n}(P_{\mathbb{N}}^{*})=P^{*}_{n}$.

Moreover denote the bootstrap sample by $\mathbf{W}^{*}_{n}:\mathcal{Z}^{\mathbb{N}}\rightarrow\mathcal{Z}^{n}$, $\mathbf{W}^{*}_{n}:=(Z^{*}_{1},\ldots,Z^{*}_{n})$, $z_{\mathbb{N}}\mapsto\mathbf{w}^{*}_{n}$, and the distribution of $\mathbf{W}^{*}_{n}$ by $\overline{P}_{n}$. The blockwise bootstrap approximation of $P_{m}$, $m\in\mathbb{N}$, is $P^{*}_{m}=\otimes_{j=1}^{m}\frac{1}{n}\sum_{i=1}^{n}\delta_{Z^{*}_{i}}$, $m\in\mathbb{N}$. Note that the sample $Z^{*}_{1},\ldots,Z^{*}_{n}$ depends and on the blocklength $b(n)$ and on the number of blocks $\ell(n)$.

Further denote the joint distribution of $\mathbf{W}_{\mathbb{N}}$, $\mathbf{W}^{*}_{\mathbb{N}}$, and $\mathbf{W}^{\prime}_{\mathbb{N}}$ by $K_{\mathbb{N}}\in\mathcal{M}(\mathcal{Z}^{\mathbb{N}}\times\mathcal{Z}^{\mathbb{N}}\times\mathcal{Z}^{\mathbb{N}})$. Then, $K_{\mathbb{N}}$ has marginal distributions $K_{\mathbb{N}}(B_{1}\times\mathcal{Z}^{\mathbb{N}}\times\mathcal{Z}^{\mathbb{N}})=P_{\mathbb{N}}(B_{1})$ for all $B_{1}\in\mathcal{B}^{\otimes\mathbb{N}}$, $K_{\mathbb{N}}(\mathcal{Z}^{\mathbb{N}}\times B_{2}\times\mathcal{Z}^{\mathbb{N}})=\overline{P}_{\mathbb{N}}(B_{2})$ for all $B_{2}\in\mathcal{B}^{\otimes\mathbb{N}}$, and $K_{\mathbb{N}}(\mathcal{Z}^{\mathbb{N}}\times\mathcal{Z}^{\mathbb{N}}\times B_{3})=P_{\mathbb{N}}^{*}(B_{3})$ for all $B_{3}\in\mathcal{B}^{\otimes\mathbb{N}}$.

Then,

[TABLE]

and therefore

[TABLE]

By assumption we have $0\leq z_{i}\leq 1$, $i\in\mathbb{N}$. Hence $Z_{i}(z_{\mathbb{N}})=z_{i}\in[0,1]$, i. e. $\mathcal{Z}=[0,1]$, which is a totally bounded metric space. Therefore the set $\textup{BL}_{1}([0,1])$ is a uniform Glivenko-Cantelli class, due to Dudley et al. (1991, Proposition 12). Similar to part I of the proof of Theorem 2.2, the blockwise bootstrap structure and the Glivenko-Cantelli property yield:

[TABLE]

Respectively, for fixed $\varepsilon>0$, for every $\delta_{0}>0$ there is $n_{1}\in\mathbb{N}$ such that for all $n\geq n_{1}$ and all $\mathbb{P}_{\mathbf{w}^{*}_{n}}\in\mathcal{M}(\mathcal{Z})$:

[TABLE]

Regard the process $G_{n}(t)=\frac{1}{\sqrt{n}}\sum_{i=1}^{n}I_{\{Z^{*}_{i}\leq t\}}-\frac{1}{\sqrt{n}}\sum_{i=1}^{n}I_{\{Z_{i}\leq t\}}$, $t\in\mathbb{R}$. Due to the assumptions on the process and on the moving block bootstrap, Theorem 2.3 in Peligrad (1998) yields the almost sure convergence in distribution to a Brownian bridge $G$:

[TABLE]

almost surely with respect to $P_{\mathbb{N}}$, $n\rightarrow\infty$, in the Skorohod topology on $D[0,1]$. Here $\longrightarrow_{\mathcal{D}}$ indicates convergence in distribution and $D[0,1]$ denotes the space of cadlag functions on $[0,1]$, for details see for example Billingsley (1999, p. 121).

This is equivalent to

[TABLE]

for all continuity points $t$ of $G$, see Billingsley (1999, (12.14), p. 124).

Multiplying by $\frac{1}{\sqrt{n}}$ yields for any fixed continuity point $t\in\mathbb{R}:$

[TABLE]

As convergence in distribution to a finite constant implies convergence in probability, see for example van der Vaart (1998, Theorem 2.7(iii)), and as $\frac{1}{\sqrt{n}}G(t)\rightarrow 0$ in probability, for all $t\in\mathbb{R}$:

[TABLE]

for all continuity points $t$ of $G$, where $\longrightarrow_{P}$ denotes the convergence in probability.

Hence, Dudley (1989, Theorem 11.12) yields the convergence of the corresponding probability measures:

[TABLE]

Respectively

[TABLE]

Define the set $B_{n}=\left\{\mathbf{w}_{n}\in\mathcal{Z}^{n}\;|\;d_{\textup{BL}}(\mathbb{P}_{\mathbf{W}^{*}_{n}},\mathbb{P}_{\mathbf{w}_{n}})\longrightarrow_{P}0,\;n\rightarrow\infty\right\}$. Hence,

[TABLE]

and, for all $\mathbf{w}_{n}\in B_{n}$, there is $n_{2,\mathbf{w}_{n}}\in\mathbb{N}$ such that for all $n\geq n_{2,\mathbf{w}_{n}}\in\mathbb{N}$:

[TABLE]

By assumption we have $0\leq z_{i}\leq 1$, $i\in\mathbb{N}$. Hence the space of probability measures $\left\{\mathbb{P}_{\mathbf{w}_{n}}\;|\;\mathbf{w}_{n}\in[0,1]^{n}\right\}$ is a subset of $\mathcal{M}([0,1])$ and therefore tight, as [0,1] is a compact space, see e. g. (Klenke, 2013, Example 13.28). Then Prohorov’s Theorem, see for example Billingsley (1999, Theorem 5.1) yields relative compactness of $\mathcal{M}([0,1],d_{\textup{BL}})$ and in particular the relative compactness of the set $\left\{\mathbb{P}_{\mathbf{w}_{n}}\;|\;\mathbf{w}_{n}\in[0,1]^{n}\right\}$. As $\mathcal{M}([0,1],d_{\textup{BL}})$ is a complete space, see Dudley (1989, Theorem 11.5.5), relative compactness equals total boundedness. That is, there exists a finite dense subset $\tilde{\mathcal{P}}$ of $\left\{\mathbb{P}_{\mathbf{w}_{n}}\;|\;\mathbf{w}_{n}\in[0,1]^{n}\right\}$ such that for all $\rho>0$ and $\mathbb{P}_{\mathbf{w}_{n}}\in\left\{\mathbb{P}_{\mathbf{w}_{n}}\;|\;\mathbf{w}_{n}\in[0,1]^{n}\right\}$ there is $\tilde{P}_{\rho}\in\tilde{\mathcal{P}}$ such that

[TABLE]

The triangle inequality yields:

[TABLE]

Define $\rho=\frac{\delta_{0}}{4}$. Then (32) yields for every $\tilde{P}_{\rho}\in\tilde{\mathcal{P}}$ the existence of an integer $n\geq n_{2,\tilde{P}}\in\mathbb{N}$ such that, for all $n\geq n_{2,\tilde{P}}$ and all $\mathbf{w}_{n}\in B_{n}$:

[TABLE]

Hence, for all $n\geq n_{2}:=\max_{\tilde{P}\in\mathcal{\tilde{P}}}\{n_{2,\tilde{P}}\}$ and for all $\mathbf{w}_{n}\in B_{n}$, we have:

[TABLE]

Due to the uniform continuity of the operator $S$, for every $\varepsilon>0$ there is $\delta_{0}>0$ such that for all $P,Q\in\mathcal{M}(\mathcal{Z})$ :

[TABLE]

Moreover, the triangle inequality yields:

[TABLE]

Again we use the relation between the Prohorov metric $\pi_{d_{H}}$ and the Ky Fan metric, Dudley (1989, Theorem 11.3.5):

[TABLE]

Due to the definition of the statistical operator $S$, this is equivalent to

[TABLE]

Due to the uniform continuity of $S$, see (35), we obtain, for all $n\geq\max\{n_{1},n_{2}\}:$

[TABLE]

The triangle inequality, (36), then yields for all $n\geq\max\{n_{1},n_{2}\}$:

[TABLE]

The equivalence between the Prohorov metric and the bounded Lipschitz metric on Polish spaces, see Huber (1981, Chapter 2, Corollary 4.3), yields the existence of $\tilde{n}_{1}$ such that for every $n\geq\tilde{n}_{1}:$

[TABLE]

And therefore

[TABLE]

For the convergence of the term in part III the same argumentation as for part I can be applied, as the assumptions on $Q_{\mathbb{N}}$ and $Q^{*}_{\mathbb{N}}$ are the same as for $P_{\mathbb{N}}$ and $\mathbb{P}^{*}_{\mathbb{N}}$. In particular for every $\varepsilon>0$ there is $\tilde{n}_{2}\in\mathbb{N}$ such that for all $n\geq\tilde{n}_{2}$:

[TABLE]

respectively

[TABLE]

Hence, (28), (37), and (38) yield, for all $n\geq\max\{\tilde{n}_{1},\tilde{n}_{2}\}$:

[TABLE]

As $\mathcal{L}_{P^{*}_{n}}(S_{n})$ and $\mathcal{L}_{Q^{*}_{n}}(S_{n})$ are random variables itself we have, due to Huber (1981, Chapter 2 Theorem 4.2, (2)$\Rightarrow$(1)), for all $n\geq\max\{\tilde{n}_{1},\tilde{n}_{2}\}$:

[TABLE]

Hence, for all $\varepsilon>0$ there is $\delta>0$ such that there is $n_{0}=\max\{\tilde{n}_{1},\tilde{n}_{2}\}\in\mathbb{N}$ such that, for all $n\geq n_{0}$:

[TABLE]

and therefore the assertion. $\square$

Proof of Theorem 2.7:

Proof of Theorem 2.7: The proof follows the same lines as the proof of Theorem 2.6 and therefore we only state the different steps. Again we start with the triangle inequality:

[TABLE]

To proof the convergence of the term in part II, we need the weak Varadarajan property of the stochastic process. Due to the definition $\alpha(\sigma(Z_{1},\ldots,Z_{i}),\sigma(Z_{i+\ell},\ldots),\mu)\leq 2$ for all $\ell\in\mathbb{N}$, $i\in\mathbb{N}$, and obviously:

[TABLE]

Hence, due to the strong stationarity of the stochastic process, we have:

[TABLE]

Now, the same argumentation as in the proof of Theorem 2.6 yields the weak Varadarajan property and therefore, for all $\varepsilon>0$,

[TABLE]

Regarding the term in part I, we use a central limit theorem for the blockwise bootstrapped empirical process by Bühlmann (1994, Corollary 1 and remark) to show its convergence. Again, regard the distribution $P_{\mathbb{N}}\in\mathcal{M}(\mathcal{Z}^{\mathbb{N}})$ and let $P^{*}_{\mathbb{N}}$ be the bootstrap approximation of $P_{\mathbb{N}}$, via the blockwise bootstrap. Define, for all $n\in\mathbb{N}$, the random variables

$\mathbf{W}_{n}:\mathcal{Z}^{\mathbb{N}}\rightarrow\mathcal{Z}^{n},\;\mathbf{W}_{n}=(Z_{1},\ldots,Z_{n})$, $z_{\mathbb{N}}\mapsto\mathbf{w}_{n}$, and

$\mathbf{W}^{\prime}_{n}:\mathcal{Z}^{\mathbb{N}}\rightarrow\mathcal{Z}^{n},\;\mathbf{W}^{\prime}_{n}=(Z^{\prime}_{1},\ldots,Z^{\prime}_{n})$, $z_{\mathbb{N}}\mapsto\mathbf{w}^{\prime}_{n}$,

such that $\mathbf{W}_{n}(P_{\mathbb{N}})=P_{n}$ and $\mathbf{W}^{\prime}_{n}(P_{\mathbb{N}}^{*})=P^{*}_{n}$.

Moreover denote the bootstrap sample by $\mathbf{W}^{*}_{n}:\mathcal{Z}^{\mathbb{N}}\rightarrow\mathcal{Z}^{n}$, $\mathbf{W}^{*}_{n}:=(Z^{*}_{1},\ldots,Z^{*}_{n})$, $z_{\mathbb{N}}\mapsto\mathbf{w}^{*}_{n}$, and the distribution of $\mathbf{W}^{*}_{n}$ by $\overline{P}_{n}$. The bootstrap approximation of $P_{m}$ is $P^{*}_{m}=\otimes_{j=1}^{m}\frac{1}{n}\sum_{i=1}^{n}\delta_{Z^{*}_{i}}=\otimes_{j=1}^{m}\mathbb{P}_{\mathbf{W}^{*}_{n}}$, $m\in\mathbb{N}$, by definition of the bootstrap procedure. Note that the sample $Z^{*}_{1},\ldots,Z^{*}_{n}$ depends and on the blocklength $b(n)$ and on the number of blocks $\ell(n)$.

Further denote the joint distribution of $\mathbf{W}_{\mathbb{N}}$, $\mathbf{W}^{*}_{\mathbb{N}}$, and $\mathbf{W}^{\prime}_{\mathbb{N}}$ by $K_{\mathbb{N}}\in\mathcal{M}(\mathcal{Z}^{\mathbb{N}}\times\mathcal{Z}^{\mathbb{N}}\times\mathcal{Z}^{\mathbb{N}})$. Then, $K_{\mathbb{N}}$ has marginal distributions $K_{\mathbb{N}}(B_{1}\times\mathcal{Z}^{\mathbb{N}}\times\mathcal{Z}^{\mathbb{N}})=P_{\mathbb{N}}(B_{1})$ for all $B_{1}\in\mathcal{B}^{\otimes\mathbb{N}}$, $K_{\mathbb{N}}(\mathcal{Z}^{\mathbb{N}}\times B_{2}\times\mathcal{Z}^{\mathbb{N}})=\overline{P}_{\mathbb{N}}(B_{2})$ for all $B_{2}\in\mathcal{B}^{\otimes\mathbb{N}}$, and $K_{\mathbb{N}}(\mathcal{Z}^{\mathbb{N}}\times\mathcal{Z}^{\mathbb{N}}\times B_{3})=P_{\mathbb{N}}^{*}(B_{3})$ for all $B_{3}\in\mathcal{B}^{\otimes\mathbb{N}}$.

Then,

[TABLE]

and therefore

[TABLE]

As $\mathcal{Z}=[0,1]^{d}$ is compact, it is in particular totally bounded. Hence the set $\textup{BL}_{1}(\mathcal{Z},d_{\mathcal{Z}})$ is a uniform Glivenko-Cantelli class, due to Dudley et al. (1991, Proposition 12). Similar to part I of the proof of Theorem 2.6, the bootstrap structure and the Glivenko-Cantelli property given above yield for arbitrary, but fixed $\varepsilon>0$:

for every $\delta_{0}>0$ there is $n_{0}\in\mathbb{N}$ such that, for all $n\geq n_{0}$ and all $\mathbb{P}_{\mathbf{w}^{*}_{n}}\in\mathcal{M}(\mathcal{Z})$,

[TABLE]

Now, regard the empirical process of $(Z_{1},\ldots,Z_{n})$. Set $\mathbf{t}=(t_{1},\ldots,t_{d})\in\mathbb{R}^{d}$. Moreover $\mathbf{t}<\mathbf{b}$ means $t_{i}<b_{i}$ for all $i\in\{1,\ldots,d\}$. Hence we can define the empirical process and the blockwise bootstrapped empirical process by

[TABLE]

Regard the process $G_{n}(\mathbf{t})=\frac{1}{\sqrt{n}}\sum_{i=1}^{n}I_{\{Z^{*}_{i}\leq\mathbf{t}\}}-\frac{1}{\sqrt{n}}\sum_{i=1}^{n}I_{\{Z_{i}\leq\mathbf{t}\}}$, $\mathbf{t}\in[0,1]^{d}$. Now, due to the assumptions on the stochastic process and on the moving block bootstrap, Bühlmann (1994, Corollary 1 and remark) yields the almost sure convergence in distribution to a Gaussian process $G$:

[TABLE]

almost surely with respect to $P_{\mathbb{N}}$, $n\rightarrow\infty$, in the (extended) Skorohod topology on $D^{d}([0,1])$.

The space $D^{d}([0,1])$ is a generalization of the space of cadlag functions on $[0,1]$, see Billingsley (1999, Chapter 12), and consists of functions $f:[0,1]^{d}\rightarrow\mathbb{R}$. A detailed description of this space and the extended Skorohod topology can be found in Straf (1972, 1969) and Bickel and Wichura (1971). The definition of the space $D^{d}([0,1])$ can, for example, be found in Bickel and Wichura (1971, Chapter 3).

Straf (1972, Lemma 5.4) yields, that the above convergence in the Skorohod topology is equivalent to the convergence for all continuity points $\mathbf{t}$ of $G$. Hence,

[TABLE]

for all continuity points $\mathbf{t}$ of $G$.

Multiplying by $\frac{1}{\sqrt{n}}$ yields, for every continuity point $\mathbf{t}$ of $G$,

[TABLE]

As convergence in distribution to a constant implies convergence in probability, see e. g. van der Vaart (1998, Theorem 2.7(iii)) and as $\frac{1}{\sqrt{n}}G(\mathbf{t})$ converges in probability to [math], for all fixed continuity points $\mathbf{t}\in[0,1]^{d}$ of $G$:

[TABLE]

This yields the convergence of the corresponding probability measures, see for example Billingsley (1995, Chapter 29) for a theory on $\mathbb{R}^{d}$:

[TABLE]

respectively

[TABLE]

As the space $[0,1]^{d}$ is compact, we can use an argumentation similar to the proof of Theorem 2.6. Then, for every $\varepsilon>0$, there is $n_{1}\in\mathbb{N}$ such that for all $n\geq n_{1}$

[TABLE]

respectively,

[TABLE]

The convergence of the term in part III follows simultaneously to part I for the distributions $Q_{\mathbb{N}}$ and $Q^{*}_{\mathbb{N}}$. Hence, for every $\varepsilon>0$, there is $n_{2}\in\mathbb{N}$ such that for all $n\geq n_{2}$

[TABLE]

The combination of (40), (41), and (42) yields for all $n\geq\max\{n_{1},n_{2}\}$:

[TABLE]

As $\mathcal{L}_{P^{*}_{n}}(S_{n})$ and $\mathcal{L}_{Q^{*}_{n}}(S_{n})$ are random variables itself we have, due to Huber (1981, Chapter 2, Theorem 4.2, (2)$\Rightarrow$(1)), for all $n\geq\max\{n_{1},n_{2}\}:$

[TABLE]

Hence, for all $\varepsilon>0$ there is $\delta>0$ such that there is $n_{0}=\max\{n_{1},n_{2}\}\in\mathbb{N}$ such that for all $n\geq n_{0}:$

[TABLE]

This yields the assertion. $\square$