Mass Volume Curves and Anomaly Ranking

TL;DR

This paper introduces the Mass Volume (MV) curve, a novel unsupervised method for ranking multivariate observations by abnormality, using statistical estimation and optimization techniques to improve anomaly detection.

Contribution

It formulates anomaly ranking as an M-estimation problem with the MV curve, providing estimation strategies, confidence regions, and bounds for the optimal scoring function.

Findings

Proposes a new MV curve-based scoring function for multivariate anomalies.

Develops a bootstrap method for confidence region estimation.

Establishes generalization bounds for the empirical MV curve.

Abstract

This paper aims at formulating the issue of ranking multivariate unlabeled observations depending on their degree of abnormality as an unsupervised statistical learning task. In the 1-d situation, this problem is usually tackled by means of tail estimation techniques: univariate observations are viewed as all the more `abnormal' as they are located far in the tail(s) of the underlying probability distribution. It would be desirable as well to dispose of a scalar valued `scoring' function allowing for comparing the degree of abnormality of multivariate observations. Here we formulate the issue of scoring anomalies as a M-estimation problem by means of a novel functional performance criterion, referred to as the Mass Volume curve (MV curve in short), whose optimal elements are strictly increasing transforms of the density almost everywhere on the support of the density. We first study the statistical estimation of the MV curve of a given scoring function and we provide a strategy to build confidence regions using a smoothed bootstrap approach. Optimization of this functional criterion over the set of piecewise constant scoring functions is next tackled. This boils down to estimating a sequence of empirical minimum volume sets whose levels are chosen adaptively from the data, so as to adjust to the variations of the optimal MV curve, while controling the bias of its approximation by a stepwise curve. Generalization bounds are then established for the difference in sup norm between the MV curve of the empirical scoring function thus obtained and the optimal MV curve.

Figures

Equations

$${\Omega }_{s,t}=\{x\in \mathcal{X}:s(x)\ge t\},t\in [-\infty ,+\infty ].$$

$$\forall (t,t^{\prime })\in {\mathbb{R}}^2,t\ge t^{\prime }\Rightarrow {\Omega }_{s,t}\subset {\Omega }_{s,t^{\prime }}$$

$${\alpha }_s(t)=\mathbb{P}\{s(X)\ge t\}\text{ and }{\lambda }_s(t)=\lambda \left(\{x\in \mathcal{X}:s(x)\ge t\}\right).$$

$$\forall t>0,{\lambda }_s(t)={\int }_{\mathcal{X}}\mathbb{I}\{x,s(x)\ge t\}dx\le {\int }_{\mathcal{X}}\frac{s(x)}{t}dx<+\infty .$$

$$\underset{\Omega }{\min }\lambda (\Omega )\text{ subject to }\mathbb{P}\{X\in \Omega \}\ge \alpha ,$$

$$\forall \alpha \in (0,1),{\lambda }^{\ast }(\alpha )\stackrel{def}{=}\lambda ({\Omega }_{\alpha }^{\ast }).$$

$$\mathbb{P}\{f(X)=t\}=0.$$

$${\mathrm{MV}}_s:\alpha \in (0,1)\mapsto {\mathrm{MV}}_s(\alpha )\stackrel{def}{=}{\lambda }_s\circ {\alpha }_s^{-1}(\alpha ).$$

$$t\in {\mathbb{R}}_{+}\mapsto \left({\alpha }_s(t),{\lambda }_s(t)\right)\in [0,1)\times [0,+\infty ).$$

$$\forall \alpha \in (0,1),{\mathrm{MV}}_{s_1}(\alpha )\le {\mathrm{MV}}_{s_2}(\alpha ).$$

$$s^{\ast }\in {\mathcal{S}}^{\ast }⟺\forall s\in \mathcal{S}\text{ and }\forall \alpha \in (0,1),{\mathrm{MV}}_{s^{\ast }}(\alpha )\le {\mathrm{MV}}_s(\alpha ).$$

$${\mathrm{MV}}_s(\alpha )-{\mathrm{MV}}^{\ast }(\alpha )\le \lambda \left({\Omega }_{\alpha }^{\ast }\Delta {\Omega }_{s,Q(s,\alpha )}\right),$$

$$s^{\ast }(x)={\int }_0^1\mathbb{I}\{x\in {\Omega }_{\alpha }^{\ast }\}d\mu (\alpha ),$$

$$\forall \alpha \in [0,1),{\mathrm{MV}}^{\ast \prime }(\alpha )=\frac{1}{Q^{\ast }(\alpha )}.$$

$${\hat{\alpha }}_s(t)\stackrel{def}{=}\frac{1}{n}\sum _{i=1}^n\mathbb{I}\{s(X_i)\ge t\}={\int }_{\mathcal{X}}\mathbb{I}\{u\ge t\}{\hat{F}}_s(du),$$

$${\widehat{\mathrm{MV}}}_s:\alpha \in [0,1)\mapsto {\lambda }_s\circ {\hat{\alpha }}_s^{-1}(\alpha ).$$

$$\forall t\in (a,b),F_s^{\prime }(t)=f_s(t)>0.$$

$$\sup_{t\in(a,b)}F_{s}(t)(1-F_{s}(t))\Bigl{|}\frac{f_{s}^{\prime}(t)}{f_{s}^{2}(t)}\Bigr{|}\leq c<+\infty\,.$$

$$\underset{t\to b^{-}}{\lim }{f}_s(t)=A<\infty .$$

$$\sup_{\alpha\in(0,1)}\alpha(1-\alpha)\Bigl{|}\frac{f_{s}^{\prime}(F_{s}^{{\dagger}}(\alpha))}{f_{s}^{2}(F_{s}^{{\dagger}}(\alpha))}\Bigr{|}\leq c<+\infty\,.$$

$$\underset{n\to +\infty }{\lim }{\widehat{\mathrm{MV}}}_s(\alpha )={\mathrm{MV}}_s(\alpha ).$$

$$\sqrt{n}\left({\widehat{\mathrm{MV}}}_s(\alpha )-{\mathrm{MV}}_s(\alpha )\right)=Z_n(\alpha )+O(n^{-1/2}\log n),$$

$$Z_n(\alpha )=\frac{{\lambda }_s^{\prime }({\alpha }_s^{-1}(\alpha ))}{f_s({\alpha }_s^{-1}(\alpha ))}{B}_n(\alpha ),\text{ for }\alpha \in [0,1-\epsilon ].$$

$$\mathbb{P}\{{\mathrm{MV}}_s\in {\mathcal{R}}_{\eta }\}\ge 1-\eta .$$

$$\mathcal{B}({\widehat{\mathrm{MV}}}_s,\nu )=\left\{g\in \mathbb{D}([0,1-\epsilon ]):\underset{\alpha \in [0,1-\epsilon ]}{\sup }|g(\alpha )-{\widehat{\mathrm{MV}}}_s(\alpha )|\le \nu \right\}.$$

$$r_n(\alpha )=\sqrt{n}({\widehat{\mathrm{MV}}}_s(\alpha )-{\mathrm{MV}}_s(\alpha )).$$

$$r_n^{Boot}=\{\sqrt{n}({\widehat{\mathrm{MV}}}_s^{Boot}(\alpha )-{\widehat{\mathrm{MV}}}_s(\alpha )){\}}_{\alpha \in [0,1)},$$

$$r_n^{\ast }=\{\sqrt{n}({\mathrm{MV}}_s^{Boot}(\alpha )-{\widetilde{\mathrm{MV}}}_s(\alpha )){\}}_{\alpha \in [0,1)},$$

$$\alpha \in [0,1)\mapsto {\widetilde{\mathrm{MV}}}_s(\alpha )={\lambda }_s\circ {\tilde{\alpha }}_s^{-1}(\alpha ).$$

$$\alpha \in [0,1)\mapsto {\mathrm{MV}}_s^{Boot}(\alpha )={\lambda }_s\circ ({\alpha }_s^{Boot}{)}^{-1}(\alpha ).$$

Full text

Keywords. Anomaly ranking, Unsupervised learning, Bootstrap, M-estimation.

1 Introduction

In a wide variety of applications, ranging from the monitoring of aircraft engines in aeronautics to non destructive control quality in the industry through fraud detection, network intrusion surveillance or system management in data centers (see for instance (Viswanathan et al., 2012)), anomaly detection is of crucial importance. In most common situations, anomalies correspond to ‘rare’ observations and must be automatically detected based on an unlabeled dataset. In practice, the very purpose of anomaly detection techniques is to rank observations by degree of abnormality/novelty, rather than simply assigning them a binary label, ‘abnormal’ vs ‘normal’. In the case of univariate observations, abnormal values are generally those which are extremes, i.e. ‘too large’ or ‘too small’ in regard to central quantities such as the mean or the median, and anomaly detection may then derive from standard tail distribution analysis: the farther in the tail the observation lies, the more ‘abnormal’ it is considered. In contrast, it is far from easy to formulate the issue in a multivariate context. In the present paper, motivated by applications such as those aforementioned, we place ourselves in the situation where (unlabeled) observations take their values in a possibly very high-dimensional space, ${\cal X}\subset\mathbb{R}^{d}$ with $d\geq 1$ say, making approaches based on nonparametric density estimation or multivariate (heavy-) tail modeling hardly feasible, if not unfeasible, due to the curse of dimensionality. In this framework, a variety of statistical techniques, relying on the concept of minimum volume set investigated in the seminal contributions of Einmahl and Mason (1992) and Polonik (1997) (see Section 2), have been developed in order to split the feature space ${\cal X}$ into two halves and decide whether observations should be considered as ‘normal’ (namely when lying in the minimum volume set $\Omega_{\alpha}\subset{\cal X}$ estimated on the basis of the dataset available) or not (when lying in the complementary set ${\cal X}\setminus\Omega_{\alpha}$) with a given confidence level $1-\alpha\in(0,1)$. One may also refer to (Scott and Nowak, 2006) and to (Koltchinskii, 1997) for closely related notions. The problem considered here is of different nature, the goal pursued is not to assign to all possible observations a label ‘normal’ vs ‘abnormal’, but to rank them according to their level of ‘abnormality’. The most natural way to define a preorder on the feature space ${\cal X}$ is to transport the natural order on the real line through some (measurable) scoring function $s:{\cal X}\rightarrow\mathbb{R}_{+}$: the ‘smaller’ the score $s(x)$, the more likely the observation $x$ is viewed as ‘abnormal’. This problem shall be here referred to as anomaly scoring. It can be somehow related to the literature dedicated to statistical depth functions in nonparametric statistics and operations research, see (Zuo and Serfling, 2000) and the references therein. Such parametric functions are generally proposed ad hoc in order to define a ‘center’ for the probability distribution of interest and a notion of distance to the latter, e.g. the concept of ‘center-outward ordering’ induced by halfspace depth in (Tukey, 1975) or (Donoho and Gasko, 1992). The angle embraced in this paper is quite different, its objective is indeed twofold: 1) propose a performance criterion for the anomaly scoring problem so as to formulate it in terms of $M$-estimation 2) investigate the accuracy of scoring rules which optimize empirical estimates of the criterion thus tailored.

Due to the global nature of the ranking problem, the criterion we promote is functional, just like the Receiver Operating Characteristic (ROC) and Precision-Recall curves in the supervised ranking setup (i.e. when a binary label, e.g. ‘normal’ vs ‘abnormal’, is assigned to the sampling data), and shall be referred to as the Mass Volume curve (MV curve in abbreviated form). The latter induces a partial preorder on the set of scoring functions: the collection of optimal elements is defined as the set of scoring functions whose MV curve is minimum everywhere. Such optimal scoring functions are proved to coincide to strictly increasing transforms of the underlying probability density almost everywhere on the support of this underlying probability density (see Section 3 for the exact definition). In the unsupervised setting, ${\rm MV}$ curve analysis is shown to play a role quite comparable to that of ${\rm ROC}$ curve analysis for supervised anomaly detection. The issue of estimating the ${\rm MV}$ curve of a given scoring function based on sampling data is then tackled and a smooth bootstrap method for constructing confidence regions is analyzed. A statistical methodology to build a nearly optimal scoring function is next described, which works as follows: first, a piecewise constant approximant of the optimal ${\rm MV}$ curve is estimated by solving a few minimum volume set estimation problems where confidence levels are chosen adaptively from the data to adjust to the variations of the optimal ${\rm MV}$ curve; second, a piecewise constant scoring function is built based on the sequence of estimated minimum volume sets. The ${\rm MV}$ curve of the scoring rule thus produced can be related to a stepwise approximant of the (unknown) optimal ${\rm MV}$ curve, which permits to establish the generalization ability of the algorithm through rate bounds in terms of $\sup$ norm in the ${\rm MV}$ space.

The rest of the article is structured as follows. Section 2 describes the mathematical framework, sets out the main notations and recalls the crucial notions related to anomaly/novelty detection on which the analysis carried out in the paper relies. Section 3 first provides an informal description of the anomaly scoring problem and then introduces the ${\rm MV}$ curve criterion, dedicated to evaluate the performance of any scoring function. The set of optimal elements is described and statistical results related to the estimation of the ${\rm MV}$ curve of a given scoring function are stated in Section 4. Statistical learning of an anomaly scoring function is then formulated as a functional $M$-estimation problem in Section 5, while Section 6 is devoted to the study of a specific algorithm for the design of nearly optimal anomaly scoring functions. Technical proofs are deferred to the Appendix section.

We finally point out that a very preliminary version of this work has been presented in the conference AISTATS 2013 (see (Clémençon and Jakubowicz, 2013)). The present article gives a better characterization of the optimal scoring functions and investigates much deeper the statistical assessment of the performance of a given scoring function. In particular, it shows how to construct confidence regions for the ${\rm MV}$ curve of a scoring function in a computationally feasible fashion, using the (smooth) bootstrap methodology for which we state a consistency result. This consistency result gives a rate of convergence which promotes the use of a smooth bootstrap approach, rather than a naive bootstrap technique. Additionally, in Section 6, the confidence levels of the minimum volume sets to be estimated are chosen in a data-driven way (instead of considering a regular subdivision of the interval $[0,1]$ fixed in advance), giving to the statistical estimation and learning procedures proposed in this paper crucial adaptivity properties, with respect to the (unknown) shape of the optimal ${\rm MV}$ curve. Eventually, we give an example showing that the nature of the problem tackled here is very different than that of density estimation and we also give a simpler formula of the derivative of the optimal ${\rm MV}$ curve (that of the underlying density) compared to the one originally given in (Clémençon and Jakubowicz, 2013).

2 Background and Preliminaries

As a first go, we start off with describing the mathematical setup and recalling key concepts in anomaly detection involved in the subsequent analysis.

2.1 Framework and Notations

Here and throughout, we suppose that we observe independent and identically distributed realizations $X_{1},\ldots,X_{n}$ of an unknown continuous probability distribution $F(dx)$, copies of a generic random variable $X$, taking their values in a (possibly very high dimensional) feature space $\mathcal{X}\subset\mathbb{R}^{d}$, with $d\geq 1$. The density of the random variable $X$ with respect to $\lambda(dx)$, the Lebesgue measure on $\mathbb{R}^{d}$, is denoted by $f$, its support $\{x\in\mathcal{X},\,f(x)>0\}$ by $\mathop{\rm supp}(f)$ and the indicator function of any event $\mathcal{E}$ by $\mathbb{I}\{\mathcal{E}\}$. For any set $\mathcal{Z}\subset\mathcal{X}$, its complementary is denoted by $\overline{\mathcal{Z}}=\mathcal{X}\setminus\mathcal{Z}$. The sup norm of any real valued function $g:\mathbb{R}^{d}\rightarrow\mathbb{R}$ is denoted by $\|g\|_{\infty}$ and the Dirac mass at any point $a$ by $\delta_{a}$. The notation $O_{\mathbb{P}}(1)$ is used to mean boundedness in probability. The quantile function $H^{{\dagger}}$ of any real valued random variable $Z$ with cumulative distribution function $H(\cdot)=\mathbb{P}(Z\leq\cdot)$ is defined by $H^{{\dagger}}(\alpha)=\inf\{t\in\mathbb{R}:H(t)\geq\alpha\}$ for all $\alpha\in(0,1)$. For any real valued random variable $Z$, the generalized inverse $G^{-1}$ of the decreasing function $G(\cdot)=\mathbb{P}(Z\geq\cdot)$ is defined by $G^{-1}(\alpha)=\inf\{t\in\mathbb{R}:G(t)\leq\alpha\}$ for all $\alpha\in(0,1)$. For any function $s:\mathcal{X}\rightarrow\mathbb{R}$, $F_{s}$ denotes the cumulative distribution function of the random variable $s(X)$. In addition, for any $\alpha\in(0,1)$, $Q(s,\alpha)=F_{s}^{{\dagger}}(1-\alpha)$ denotes the quantile at level $1-\alpha$ of the distribution of $s(X)$. We also set $Q^{*}(\alpha)=Q(f,\alpha)$ for all $\alpha\in(0,1)$. Finally, constants in inequalities are denoted by either $C$ or $c$ and may vary at each occurrence.

A natural way of defining preorders on ${\cal X}$ is to map its elements onto $\mathbb{R}_{+}$ and use the natural order on the real half-line.

Definition 1**.**

(Scoring function)* A scoring function is any measurable function $s:\mathcal{X}\rightarrow\mathbb{R}_{+}$ that is integrable with respect to the Lebesgue measure.*

The set of all scoring functions is denoted by $\mathcal{S}$ and we denote the level sets of any scoring function $s\in\mathcal{S}$ by:

[TABLE]

Observe that the family is decreasing (for the inclusion, as $t$ increases from $-\infty$ to $+\infty$):

[TABLE]

and that $\lim_{t\rightarrow+\infty}\Omega_{s,t}=\emptyset$ and $\lim_{t\rightarrow-\infty}\Omega_{s,t}={\cal X}$111Recall that a sequence $(A_{n})_{n\geq 1}$ of subsets of an ensemble $E$ is said to converge if and only if $\lim\sup A_{n}=\lim\inf A_{n}$. In such a case, one defines its limit, denoted by $\lim A_{n}$ as $\lim\sup A_{n}=\lim\inf A_{n}$..

The following quantities shall also be used in the sequel. For any scoring function $s$ and threshold level $t\geq 0$, define:

[TABLE]

The quantity $\alpha_{s}(t)$ is referred to as the mass of the level set $\Omega_{s,t}$, while $\lambda_{s}(t)$ is generally termed the volume (with respect to the Lebesgue measure).

Remark 1**.**

The integrability of a scoring function (see Definition 1) implies that the volumes are finite on $\mathbb{R}_{+}^{*}$: for any $s\in\mathcal{S}$,

[TABLE]

Incidentally, we point out that, in the specific case $s=f$, the set $\Omega_{f,t}=\{x\in\mathcal{X},f(x)\geq t\}$ is the density contour cluster of the density function $f$ at level $t$, see (Polonik, 1995) for instance. Such a set is also referred to as a density level set. Observe in addition that, using the terminology introduced in (Liu et al., 1999), $\Omega_{s,t}$ is the region enclosed by the contour of depth $t$ when $s$ is a depth function.

2.2 Minimum Volume Sets

The notion of minimum volume sets has been introduced and investigated in the seminal contributions of Einmahl and Mason (1992) and Polonik (1997) in order to describe regions where a multivariate random variable $X$ takes its values with highest/smallest probability. Let $\alpha\in(0,1)$, a minimum volume set $\Omega^{*}_{\alpha}$ of mass at least $\alpha$ is any solution of the constrained minimization problem

[TABLE]

where the minimum is taken over all measurable subsets $\Omega$ of ${\cal X}$. Application of this concept includes in particular novelty/anomaly detection: for large values of $\alpha$, abnormal observations (outliers) are those which belong to the complementary set ${\cal X}\setminus\Omega^{*}_{\alpha}$ . In the continuous setting, it can be shown that there exists a threshold value equal to $Q^{*}(\alpha)$ such that the level set $\Omega_{f,Q^{*}(\alpha)}$ is a solution of the constrained optimization problem above. The (generalized) quantile function is then defined by:

[TABLE]

The definition of $\lambda^{*}$ can be extended to the interval $[0,1]$ by setting $\lambda^{*}(0)=0$ and $\lambda^{*}(1)=\lambda(\mathop{\rm supp}(f))\leq\infty$. The following assumptions shall be used in the subsequent analysis.

1. (A1)

The density $f$ is bounded: $\|f\|_{\infty}<+\infty$. 2. (A2)

The density $f$ has no flat parts, i.e. for any $t>0$,

[TABLE]

Under the assumptions above, for any $\alpha\in(0,1)$, there exists a unique minimum volume set $\Omega^{*}_{\alpha}$ (equal to $\Omega_{f,Q^{*}(\alpha)}$ up to subsets of null $\lambda$-measure), whose mass is equal to $\alpha$ exactly. Additionally, the mapping $\lambda^{*}$ is continuous on $(0,1)$ and uniformly continuous on $[0,1-\varepsilon]$ for all $\varepsilon\in(0,1)$ (when the support of $F(dx)$ is compact, uniform continuity holds on the whole interval $[0,1]$).

From a statistical perspective, estimates $\widehat{\Omega}_{\alpha}$ of minimum volume sets are built by replacing the unknown probability distribution $F$ by its empirical version $\widehat{F}=(1/n)\sum_{i=1}^{n}\delta_{X_{i}}$ and restricting optimization to a collection $\mathcal{A}$ of borelian subsets of ${\cal X}$, supposed rich enough to include all density level sets (or reasonable approximations of the latter). In (Polonik, 1997), functional limit results are derived for the generalized empirical quantile process $\{\lambda(\widehat{\Omega}^{*}_{\alpha})-\lambda^{*}(\alpha)\}$ under certain assumptions for the class $\mathcal{A}$ (stipulating in particular that $\mathcal{A}$ is a Glivenko-Cantelli class for $F(dx)$). In (Scott and Nowak, 2006), it is proposed to replace the level $\alpha$ by $\alpha-\phi_{n}$ where $\phi_{n}$ plays the role of tolerance parameter. The latter should be chosen of the same order as the supremum $\sup_{\Omega\in\mathcal{A}}|\widehat{F}(\Omega)-F(\Omega)|$ roughly, complexity of the class $\mathcal{A}$ being controlled by the VC dimension or by means of the concept of Rademacher averages, so as to establish rate bounds at $n<+\infty$ fixed.

Alternatively, so-termed plug-in techniques, consisting in computing first an estimate $\widehat{f}$ of the density $f$ and considering next level sets $\Omega_{\widehat{f},t}$ of the resulting estimator have been investigated in several papers, among which (Cavalier, 1997; Tsybakov, 1997; Rigollet and Vert, 2009; Cadre, 2006; Cadre et al., 2013). Such an approach however yields significant computational issues even for moderate values of the dimension, inherent to the curse of dimensionality phenomenon.

3 Ranking Anomalies

In this section, the issue of scoring observations depending on their level of novelty/abnormality is first described in an informal manner and next formulated quantitatively, as a functional optimization problem, by means of a novel concept, termed the Mass Volume curve.

3.1 Overall Objective

The problem considered in this article is to learn a scoring function $s$, based on training data $X_{1},\ldots,X_{n}$, so as to describe the extremal behavior of the (high-dimensional) random vector $X$ by that of the univariate variable $s(X)$, which can be summarized by its tail behavior near [math]: hopefully, the smaller the score $s(x)$, the more abnormal/rare the observation $x$ should be considered. Hence, an optimal scoring function should naturally permit to rank observations $x$ by increasing order of magnitude of $f(x)$ (see Section 3.2 for the precise definition of the set of optimal scoring functions). The preorder on ${\cal X}$ induced by such an optimal scoring function could then be used to rank observations by their degree of abnormality: for any optimal scoring function $s$, an observation $x^{\prime}$ is more abnormal than an observation $x$ if and only if $s(x^{\prime})<s(x)$.

3.2 A Functional Criterion: the Mass Volume Curve

We now introduce the concept of Mass Volume curve and shows that it is a natural criterion to evaluate the accuracy of decision rules in regard to anomaly scoring.

Definition 2**.**

(True Mass Volume curve)* Let $s\in{\cal S}$. Its Mass Volume curve (${\rm MV}$ curve in abbreviated form) with respect to the probability distribution of the random variable $X$ is the plot of the function*

[TABLE]

If the scoring function $s$ is upper bounded, $\alpha_{s}^{-1}(0)$ exists and ${\rm MV}_{s}$ is defined on $[0,1)$.

Remark 2**.**

(Parametric Mass Volume Curve)* If $\alpha_{s}$ is invertible, and therefore $\alpha_{s}^{-1}$ is the ordinary inverse of $\alpha_{s}$, the Mass Volume curve of the scoring function $s$ can also be defined as the parametric curve:*

[TABLE]

Remark 3**.**

(Connections to ROC/concentration analysis)* We point out that the curve $\alpha\in(0,1)\mapsto 1-\lambda_{s}\circ\alpha_{s}^{-1}(1-\alpha)$ resembles a receiver operating characteristic (ROC) curve of the test/diagnostic function $s$ (see (Egan, 1975)), except that the distribution under the alternative hypothesis is not a probability measure but the image of Lebesgue measure on ${\cal X}$ by the function $s$, while the distribution under the null hypothesis is the probability distribution of the random variable $s(X)$. Hence, the curvature of the ${\rm MV}$ graph somehow measures the extent to which the spread of the distribution of $s(X)$ differs from that of a uniform distribution. Observe also that, in the case where the support of $F(dx)$ coincides with the unit square $[0,1]^{d}$, the ${\rm MV}$ curve of any scoring function $s\in\mathcal{S}$ corresponds to the concentration function $(GF^{-1})_{\mathcal{C}_{\mathcal{S}}}$ introduced in (Polonik, 1999), where $G(dx)$ is the uniform probability distribution on $[0,1]^{d}$ and $\mathcal{C}_{\mathcal{S}}=\{\Omega_{s,t}:\;t\geq 0\}$, to compare the concentration of $F(dx)$ with that of $G(dx)$ with respect to the class of sublevel sets of the function $s$. Notice finally that, when $s$ is a depth function, ${\rm MV}_{s}$ is a scale curve, as defined in (Liu et al., 1999).*

This functional criterion induces a partial order over the set of all scoring functions. Let $s_{1}$ and $s_{2}$ be two scoring functions on ${\cal X}$, the ordering provided by $s_{1}$ is better than that induced by $s_{2}$ when

[TABLE]

Typical ${\rm MV}$ curves are illustrated in Fig. 1. A desirable ${\rm MV}$ curve increases slowly and rises near $1$, just like the lowest curve of Fig. 1. This corresponds to the situation where the distribution of the random variable $X$ is much concentrated around its modes and the highest values (respectively, the lowest values) taken by $s$ are located near the modes of $F(dx)$ (respectively, in the tail region of $F(dx)$). The ${\rm MV}$ curve of the scoring function $s$ is then close to the right lower corner of the Mass Volume space. We point out that, in certain situations, some parts of the MV curve may be of interest solely, corresponding to large values of $\alpha$ when focus is on extremal observations (the tail region of the random variable $X$) and to small values of $\alpha$ when modes of the underlying distribution are investigated.

We define the set of optimal scoring functions $\mathcal{S}^{*}$ as follows. We recall here that for any set $\mathcal{Z}\subset\mathcal{X}$, its complementary is denoted by $\overline{\mathcal{Z}}=\mathcal{X}\setminus\mathcal{Z}$.

Definition 3**.**

The set of optimal scoring functions $\mathcal{S}^{*}$ is the set of scoring functions $s\in\mathcal{S}$ such that there exist $\mathcal{Z}\subset\mathcal{X}$ and a function $T:Imf\rightarrow\mathbb{R}^{+}$ such that

1. (i)

$\lambda(\overline{\mathcal{Z}}\cap\mathop{\rm supp}(f))=0$, 2. (ii)

For all $x\in\mathcal{Z}\cap\mathop{\rm supp}(f)$, $s(x)=T\circ f(x)$, 3. (iii)

$T_{|f(\mathcal{Z}\cap\mathop{\rm supp}(f))}$* is strictly increasing,* 4. (iv)

For $\lambda$-almost all $x\in\overline{\mathop{\rm supp}(f)}$, $F_{s}(s(x))=0$.

One can note that, as expected, the density $f$ and strictly increasing transforms of the density belong to $\mathcal{S}^{*}$. More generally and roughly speaking, an optimal scoring function $s\in\mathcal{S^{*}}$ is a strictly increasing transform of the density $\lambda$-almost everywhere on the support of the density. Note also that on the one hand, if $a_{s}=\sup\{t,F_{s}(t)=0\}$, thanks to $(i)$, $(ii)$ and $(iii)$, we have $F_{s}(s(x))=F_{f}(f(x))>0$ for almost all $x\in\mathop{\rm supp}(f)$ (because $\mathop{\rm supp}(f)=\{x,F_{f}(f(x))>0\}$ up to subsets of null $\lambda$ measure as shown in Lemma 1). Therefore $s\geq a_{s}$ $\lambda$-almost everywhere on $\mathop{\rm supp}(f)$. On the other hand, thanks to $(iv)$, $F_{s}(s(x))=0$ for almost all $x\in\overline{\mathop{\rm supp}(f)}$ and therefore $s\leq a_{s}$ $\lambda$-almost everywhere on $\overline{\mathop{\rm supp}(f)}$. Thus the values of $s$ on $\overline{\mathop{\rm supp}(f)}$ are $\lambda$-almost everywhere lower than the values of $s$ on $\mathop{\rm supp}(f)$, where $s$ is almost everywhere a strictly increasing transform of the density $f$.

The result below shows that optimal scoring functions are those whose ${\rm MV}$ curves are minimum everywhere.

Proposition 1**.**

(Optimal ${\rm MV}$ curve)* Let assumptions (A1) and (A2) be fulfilled. The elements of the class ${\cal S}^{*}$ have the same ${\rm MV}$ curve, equal to ${\rm MV}_{f}$, and provide the best possible ordering of ${\cal X}$’s elements in regard to the ${\rm MV}$ curve criterion:*

[TABLE]

In addition, we have: $\forall s\in{\cal S}$ and $\forall\alpha\in(0,1)$,

[TABLE]

where ${\rm MV}^{*}(\alpha)={\rm MV}_{f}(\alpha)$ for all $\alpha\in(0,1)$ and where $\Delta$ denotes the symmetric difference.

From now on we will thus use ${\rm MV}^{*}={\rm MV}_{f}$ to denote the ${\rm MV}$ curve of elements of $\mathcal{S}^{*}$. The proof of Proposition 1 can be found in Appendix A. Incidentally, notice that, equipped with the notations introduced in 2.2, we have $\lambda^{*}(\alpha)={\rm MV}^{*}(\alpha)$ for all $\alpha\in(0,1)$. We also point out that bound (3) reveals that the pointwise difference between the optimal ${\rm MV}$ curve and that of a scoring function candidate $s$ is controlled by the error made in recovering the specific minimum volume set $\Omega^{*}_{\alpha}$ through $\Omega_{s,Q(s,\alpha)}$.

Remark 4**.**

In the framework we develop, anomaly scoring boils down to recovering the decreasing collection of all level sets of the density function $f$, $\{\Omega^{*}_{\alpha}:\;\alpha\in(0,1)\}$, without necessarily disposing of the corresponding levels. Indeed, one may check that any scoring function of the form

[TABLE]

where $\mu(d\alpha)$ is an arbitrary finite positive measure dominating the Lebesgue measure on $(0,1)$, belongs to ${\cal S}^{*}$. Observe that $F_{f}\circ f$ corresponds to the case where $\mu$ is chosen to be the Lebesgue measure on $(0,1)$ in Eq. (4). The anomaly scoring problem can be thus cast as an overlaid collection of minimum volume set estimation problems. This observation shall turn out to be very useful when designing practical statistical learning strategies in Section 6.

We point out that the optimal ${\rm MV}$ curve provides a measure of the mass concentration of the random variable $X$: the lower the curve ${\rm MV}^{*}$, the more concentrated the distribution $f(x)\lambda(dx)$. Indeed, the excess mass functional introduced in (Muller and Sawitzki, 1991) can be expressed as $t\geq 0\mapsto E(t)=\alpha_{f}(t)-t\lambda_{f}(t)$. Hence, we have $E(Q^{*}(\alpha))=\alpha-Q^{*}(\alpha)MV^{*}(\alpha)$ for all $\alpha\in(0,1)$. One may refer to (Polonik, 1995) for results related to the statistical estimation of density contour clusters and applications to multimodality testing using the excess mass functional.

Example 1**.**

(Univariate Gaussian distribution)* Let $X$ be random variable with Gaussian distribution $\mathcal{N}(0,1)$, we have ${\rm MV}^{*}(\alpha)=2\Phi^{-1}((1+\alpha)/2)$, where $\Phi$ is the cumulative distribution function of $X$.*

Example 2**.**

(Multivariate Gaussian distribution)* Let $X\in\mathbb{R}^{d}$ be a multivariate random variable with Gaussian distribution $\mathcal{N}(0,\mathbf{\Sigma})$. We assume that $\Sigma$ is a diagonal matrix and we denote by $a_{1},\dots,a_{d}\in\mathbb{R}^{+*}$ its diagonal coefficients. The density level set $\{x,f(x)\geq t\}$ is exactly the set $\{x,x^{T}\Sigma^{-1}x\leq c_{t}\}$ with $c_{t}>0$. It is well known that $X^{T}\Sigma^{-1}X\sim\chi^{2}_{d}$, the $\chi^{2}$ distribution with $d$ degrees of freedom. Thus $\alpha_{f}^{-1}(\alpha)=c_{t}=\chi^{2}_{d}(\alpha)$, $\chi^{2}_{d}(\alpha)$ denoting the quantile of order $\alpha$ of the $\chi^{2}_{d}$ distribution. The set $\{x,x^{T}\Sigma^{-1}x\leq\chi^{2}_{d}(\alpha)\}$ is exactly the ellipsoid with semi-principal axes of length $a_{1}\sqrt{\chi^{2}_{d}(\alpha)},\dots,a_{d}\sqrt{\chi^{2}_{d}(\alpha)}$. The formula of the volume of an ellipsoid gives ${\rm MV}^{*}(\alpha)=\pi^{d/2}/\Gamma(d/2+1)(\chi^{2}_{d}(\alpha))^{d/2}\prod_{i=1}^{d}a_{i}$ where $\Gamma$ is the gamma function.*

The following result reveals that the optimal ${\rm MV}$ curve is convex and provides a closed analytical form for its derivative. The proof is given in Appendix A.

Proposition 2**.**

(Convexity and derivative)* Suppose that assumptions (A1) and (A2) are satisfied. Then, as $f$ is bounded, the optimal curve ${\rm MV}^{*}$ is defined on $[0,1)$. Furthermore,*

1. (i)

$\alpha\in[0,1)\mapsto{\rm MV}^{*}(\alpha)$* is convex.* 2. (ii)

Let $F_{f}$ be the cumulative distribution function of the random variable $f(X)$ and let $a=\lim_{\alpha\rightarrow 0^{+}}F_{f}^{{\dagger}}(\alpha)=\sup\{t,F_{f}(t)=0\}\geq-\infty$. If $F_{f}$ is invertible on $(a,F_{f}^{{\dagger}}(1)]$ then ${\rm MV}^{*}$ is differentiable on $[0,1)$ and

[TABLE]

The convexity of ${\rm MV}^{*}$ might be explained as follows: when considering density level sets with increasing probabilities $\alpha$, their volumes $\lambda(\Omega^{*}_{\alpha})$ increase as well, but they increase more and more quickly since the mass of the distribution becomes less and less concentrated.

Elementary properties of ${\rm MV}$ curves are summarized in the following proposition.

Proposition 3**.**

(Properties of ${\rm MV}$ curves)* For any $s\in{\cal S}$, the following assertions hold true.*

1. (i)

Invariance.* For any strictly increasing function $\psi:\mathbb{R}_{+}\rightarrow\mathbb{R}_{+}$, we have ${\rm MV}_{s}={\rm MV}_{\psi\circ s}$.* 2. (ii)

Monotonicity.* The mapping $\alpha\in(0,1)\mapsto{\rm MV}_{s}(\alpha)$ is increasing.*

Assertion $(i)$ may be proved using the following property of the quantile function (see e.g. Property 2.3 in (Embrechts and Hofert, 2013)): for any cumulative distribution function $H$, any $z\in\mathbb{R}$ and any $\alpha\in(0,1)$, $z\geq H^{{\dagger}}(\alpha)$ if and only if $H(z)\geq\alpha$. Assertion $(ii)$ derives from the fact that the quantile function $F_{s}^{{\dagger}}$ is increasing. Details are left to the reader.

4 Statistical Estimation

In practice, ${\rm MV}$ curves are unknown, just like the probability distribution of the random variable $X$, and must be estimated based on the observed sample $X_{1},\dots,X_{n}$. Replacing the mass of each level set by its statistical counterpart in Definition 2 leads to define the notion of empirical ${\rm MV}$ curve. We set, for all $t\geq 0$,

[TABLE]

where $\widehat{F}_{s}=(1/n)\sum_{i\leq n}\delta_{s(X_{i})}$ denotes the empirical distribution of the $s(X_{i}),1\leq i\leq n$. Notice that $\widehat{\alpha}_{s}$ takes its values in the set $\{k/n:\;\;k=0,\dots,n\}$.

Definition 4**.**

(Empirical ${\rm MV}$ curve)* Let $s\in{\cal S}$. By definition, the empirical ${\rm MV}$ curve of $s$ is the graph of the (piecewise constant) function*

[TABLE]

Remark 5**.**

(On volume estimation). Except for very specific choices of the scoring function $s$ (e.g. when $s$ is piecewise constant and the volumes of the subsets of $\mathcal{X}$ on which $s$ is constant can be explicitly computed), no closed analytic form for the volume $\lambda_{s}(t)$ is available in general and Monte-Carlo procedures should be used to estimate it (which may be practically challenging in a high-dimensional setting). Computation of volumes in high dimensions is an active topic of research (Lovász and Vempala, 2006) and for simplicity, this is not taken into account in the subsequent analysis.

In order to obtain a smoothed version $\widetilde{\alpha}_{s}(t)$, a typical strategy consists in replacing the empirical distribution estimate $\widehat{F}_{s}$ involved in (5) by the continuous distribution $\widetilde{F}_{s}$ with density $\widetilde{f}_{s}(t)=(1/n)\sum_{i=1}^{n}K_{h}(s(X_{i})-t)$, with $K_{h}(t)=h^{-1}K(t/h)$ where $K\geq 0$ is a regularizing Parzen-Rosenblatt kernel (i.e. a bounded square integrable function such that $\int K(v)dv=1$) and $h>0$ is the smoothing bandwidth (see for instance (Wand and Jones, 1994)).

4.1 Consistency and Asymptotic Normality

The theorem below reveals that, under mild assumptions, the empirical ${\rm MV}$ curve is a consistent and asymptotically Gaussian estimate of the ${\rm MV}$ curve, uniformly over any subinterval of $[0,1)$. It involves the assumptions listed below.

1. (A3)

The scoring function $s$ is bounded: $\|s\|_{\infty}<+\infty$. 2. (A4)

The random variable $s(X)$ has a continuous cumulative distribution function $F_{s}$. Let $a=\sup\{t\in\mathbb{R},\,F_{s}(t)=0\}\geq-\infty$ and $b=\inf\{t\in\mathbb{R},\,F_{s}(t)=1\}\leq+\infty$. The distribution function $F_{s}$ is twice differentiable on $(a,b)$ and

[TABLE] 3. (A5)

There exists $c>0$ such that

[TABLE] 4. (A6)

$f_{s}$ has a limit $A>0$ when $t$ tends towards $b$ from the left:

[TABLE] 5. (A7)

The mapping $\lambda_{s}$ is of class $\mathcal{C}^{2}$.

Assumption (A3) implies that $F_{s}^{{\dagger}}(1)$ exists and $F_{s}^{{\dagger}}(1)=b$. Note also that if $s$ is not bounded, one can always consider $s^{\prime}=\arctan\circ s$. Indeed $s^{\prime}$ takes its values in $[0,\pi/2)$ and as $\arctan$ is strictly increasing ${\rm MV}_{s}={\rm MV}_{s^{\prime}}$ thanks to Proposition 3. Assumptions (A4) and (A5) are common assumptions for the strong approximation of the quantile process (see for instance (Csörgő and Révész, 1978)). Condition (6) and assumption (A5) are respectively equivalent to $f_{s}(F_{s}^{{\dagger}}(\alpha))>0$ for all $\alpha\in(0,1)$ and

[TABLE]

Eventually, assumption (A6) is equivalent to: $f_{s}$ has a limit $A>0$ when $x$ tends towards $b$. The density $f_{s}$ can therefore be extended by continuity to the interval $(a,b]$. As $A>0$, (6) can then be replaced by $f_{s}(t)>0$ for all $t\in(a,b]$ which also gives $f_{s}(F_{s}^{{\dagger}}(\alpha))>0$ for all $\alpha\in(0,1]$. One can also note that $F_{s}$ is thus invertible on $(a,b]$ with inverse $F_{s}^{{\dagger}}$.

In order to state part $(ii)$ of the following theorem, we assume that the probability space is rich enough in the sense that an infinite sequence of Brownian bridges can be defined on it.

Theorem 1**.**

Let $\varepsilon\in(0,1]$ and $s\in{\cal S}$. Assume that assumptions (A3)-(A7) are fulfilled. The following assertions hold true.

1. (i)

(Consistency)* With probability one, we have uniformly over $[0,1-\varepsilon]$:*

[TABLE] 2. (ii)

(Strong approximation)* There exists a sequence of Brownian bridges $\{B_{n}(\alpha),\alpha\in[0,1]\}_{n\geq 1}$ such that we almost surely have, uniformly over the compact interval $[0,1-\varepsilon]$: as $n\rightarrow\infty$,*

[TABLE]

where

[TABLE]

The technical proof is given in Appendix A. One can note from the proof of assertion $(i)$ that this assertion does not require assumption (A5) to be satisfied. The proof of assertion $(ii)$ relies on standard strong approximation results for the quantile process (Csörgő and Révész, 1978, 1981; Csörgő, 1983). Assertion $(ii)$ means that the fluctuation process $\{\sqrt{n}(\widehat{{\rm MV}}_{s}(\alpha)-{\rm MV}_{s}(\alpha)),\alpha\in[0,1-\varepsilon]\}$ converges in the space of càd-làg functions on $[0,1-\varepsilon]$ equipped with the sup norm, to the law of a Gaussian stochastic process $\{Z_{1}(\alpha),\alpha\in[0,1-\varepsilon]\}$.

Remark 6**.**

Assumption (A6) is required in order to state the results of assertions $(i)$ and $(ii)$ on the interval $[0,1-\varepsilon]$. However this assumption is restrictive as it implies that $f_{s}$ is discontinuous at $b$ since $A>0$ whereas $f_{s}(t)=0$ for all $t>b$. Instead of assumption (A6), if one assumes that $f_{s}$ is decreasing on an interval to the left of $b$, then the result of assertion $(ii)$ can be obtained on the interval $(0,1-\varepsilon]$ with the same rate of convergence if $c<2$ and with the same rate of convergence up to $\log n$ and $\log\log n$ factors if $c\geq 2$. In this case, assertion $(i)$ also holds on $(0,1-\varepsilon]$ (see for instance (Csörgő and Révész, 1978)).

Remark 7**.**

(Asymptotic normality)* It results from assertion $(ii)$ in Theorem 1 that, for any $\alpha\in(0,1)$, the pointwise estimator $\widehat{{\rm MV}}_{s}(\alpha)$ is asymptotically Gaussian under assumptions (A3)-(A7). For all $\alpha\in(0,1)$ $\sqrt{n}(\widehat{{\rm MV}}_{s}(\alpha)-{\rm MV}_{s}(\alpha))$ converges in distribution towards $\mathcal{N}(0,\sigma^{2}_{s})$, as $n\rightarrow+\infty$, with $\sigma^{2}_{s}=\alpha(1-\alpha)(\lambda_{s}^{\prime}(\alpha_{s}^{-1}(\alpha))/f_{s}(\alpha_{s}^{-1}(\alpha)))^{2}$.*

4.2 Confidence Regions in the Mass Volume Space

The true ${\rm MV}$ curve of a given scoring function is unknown in practice and its performance must be statistically assessed based on a data sample. Beyond consistency of the empirical curve in sup norm and the asymptotic normality of the fluctuation process, we now tackle the question of constructing confidence bands in the ${\rm MV}$ space.

Definition 5**.**

Based on a sample $\mathcal{D}_{n}=(X_{1},\;\ldots,\;X_{n})$, a (random) confidence region for the ${\rm MV}$ curve of a given scoring function $s\in{\cal S}$ at confidence level $\eta\in(0,1)$ is any borelian set $\mathcal{R}_{\eta}\subset[0,1]\times\mathbb{R}_{+}$ of the ${\rm MV}$ space (possibly depending on $\mathcal{D}_{n}$) that covers the curve ${\rm MV}_{s}$ with probability larger than $1-\eta$:

[TABLE]

In practice, confidence regions shall be of the form of balls in the Skorohod’s space $\mathbb{D}([0,1-\varepsilon])$ of càd-làg functions on $[0,1-\varepsilon]$ with respect to the $\sup$ norm and with the estimate $\widehat{{\rm MV}}_{s}$ introduced in Definition 4 as center for some fixed $\varepsilon\in(0,1)$:

[TABLE]

Constructing confidence regions based on the approximation stated in Theorem 1 would require to know the density $f_{s}$. Hence a bootstrap approach (Efron, 1979) should be preferred. Following in the footsteps of Silverman and Young (1987), it is recommended to implement a smoothed bootstrap procedure. The asymptotic validity of such a resampling method derives from a strong approximation result similar to the one of Theorem 1. Let $r_{n}$ be the fluctuation process defined for all $\alpha\in[0,1-\varepsilon]$ by

[TABLE]

The bootstrap approach suggests to consider, as an estimate of the law of the fluctuation process $r_{n}$, the conditional law given the original sample $\mathcal{D}_{n}=(X_{1},\ldots,X_{n})$ of the naive bootstrapped fluctuation process

[TABLE]

where, given $\mathcal{D}_{n}$, $\widehat{{\rm MV}}^{Boot}_{s}$ is the empirical ${\rm MV}$ curve of the scoring function $s$ based on a sample of i.i.d. random variables with distribution $\widehat{F}_{s}=(1/n)\sum_{i\leq n}\delta_{s(X_{i})}$. The difficulty is twofold. First, the target is a distribution on a path space, namely a subspace of the Skorohod’s space $\mathbb{D}([0,1-\varepsilon])$ equipped with the $\sup$ norm. Second, $r_{n}$ is a functional of the quantile process $\{\widehat{F}_{s}^{{\dagger}}(\alpha)\}_{\alpha\in[\varepsilon,1]}$. The naive bootstrap, which consists in resampling from the raw empirical distribution $\widehat{F}_{s}$, generally provides bad approximations of the distribution of empirical quantiles: the rate of convergence for a given quantile is indeed of order $O_{\mathbb{P}}(n^{-1/4})$ (Falk and Reiss, 1989) whereas the rate of the Gaussian approximation is $O(n^{-1/2}\log n)$ (see (25) in Appendix A). The same phenomenon may be naturally observed for ${\rm MV}$ curves. In a similar manner to what is usually recommended for empirical quantiles, a smoothed version of the bootstrap algorithm shall be implemented in order to improve the approximation rate of the distribution of $\sup_{\alpha\in[0,1-\varepsilon]}|r_{n}(\alpha)|$, namely, to resample the data from a smoothed version $\widetilde{F}_{s}$ of the empirical distribution $\widehat{F}_{s}$. We thus consider the smooth boostrapped fluctuation process

[TABLE]

where, given $\mathcal{D}_{n}$, ${\rm MV}^{Boot}_{s}=\lambda_{s}\circ(\alpha^{Boot}_{s})^{-1}$ is the empirical ${\rm MV}$ curve of the scoring function $s$ based on a sample of i.i.d. random variables with distribution $\widetilde{F}_{s}$ and where $\widetilde{\rm MV}_{s}=\lambda_{s}\circ\widetilde{\alpha}_{s}^{-1}$ is the smooth version of the empirical ${\rm MV}$ curve, $\widetilde{\alpha}_{s}^{-1}$ being the generalized inverse of $\widetilde{\alpha}_{s}$. The algorithm for building a confidence band at level $1-\eta$ in the ${\rm MV}$ space from sampling data $\mathcal{D}_{n}=\{X_{i}:\;i=1,\;\ldots,\;n\}$ is described in Algorithm 1.

Before turning to the theoretical analysis of this algorithm, its description calls a few comments. From a computational perspective, the smoothed bootstrap distribution $\mathbb{P}^{*}(\sup_{\alpha\in[\varepsilon,1-\varepsilon]}|r^{*}_{n}(\alpha)|\leq\cdot)$ must be approximated in its turn, by means of a Monte-Carlo approximation scheme. Based on the $N$ bootstrap fluctuation processes obtained, $r_{n}^{*(j)}$ with $j=1,\;\ldots,\;N$, the radius $\nu_{\eta}$ then coincides with the empirical quantile at level $1-\eta$ of the statistical population $\{\sup_{\alpha\in[\varepsilon,1-\varepsilon]}|r_{n}^{*(j)}(\alpha)|:\;j=1,\;\ldots,\;N\}$. Concerning the number of bootstrap replications, picking $N=n$ does not modify the rate of convergence. However, choosing $N$ of magnitude comparable to $n$ so that $(1+N)(1-\eta)$ is an integer may be more appropriate: the $(1-\eta)$-quantile of the approximate bootstrap distribution is then uniquely defined and this does not impact the rate of convergence neither (Hall, 1986).

4.2.1 Bootstrap consistency

The next theorem reveals the asymptotic validity of the bootstrap estimate proposed above where we assume that the smoothed version $\widetilde{F}_{s}$ of the distribution function $\widehat{F}_{s}$ is computed at step 1 of Algorithm 1 using a kernel $K_{h_{n}}$. It requires the following assumptions.

1. (B1)

The density $f_{s}$ is bounded and of class $\mathcal{C}^{3}$. 2. (B2)

The bandwidth $h_{n}$ decreases towards [math] as $n\rightarrow+\infty$ in a way that $nh_{n}\rightarrow+\infty$, $nh_{n}/\log(h_{n}^{-1})\rightarrow+\infty$ and $\log(h_{n}^{-1})/\log\log n\rightarrow+\infty$. 3. (B3)

The kernel $K$ has finite support and satisfies the following conditions: $\int K(y)dy=1$, $\int yK(y)dy=0$ and $\int y^{2}K(y)dy<+\infty$. 4. (B4)

The kernel $K$ is such that $\|K\|_{2}<+\infty$ and is of the form $K(y)=\Phi_{1}(P_{1}(y))$, $P_{1}$ being a polynomial and $\Phi_{1}$ a bounded real function of bounded variation. 5. (B5)

The kernel $K$ is differentiable with derivative $K^{\prime}$ such that $\|K^{\prime}\|_{2}<+\infty$ and of the form $K^{\prime}(y)=\Phi_{2}(P_{2}(y))$, $P_{2}$ being a polynomial and $\Phi_{2}$ a bounded real function of bounded variation. 6. (B6)

The bandwidth $h_{n}$ is such that $\log(h_{n}^{-1})/(nh_{n}^{3})$ tends to 0 as $n\rightarrow+\infty$.

Assumptions (B1) and (B3) are needed to control the biases $\sup_{\mathbb{R}}|\mathbb{E}[\widetilde{f}_{s}]-f_{s}|$ and $\sup_{\mathbb{R}}|\mathbb{E}[\widetilde{f_{s}}^{\prime}]-f_{s}^{\prime}|$. Assumption (B2) on the bandwidth $h_{n}$ and assumption (B4) (respectively assumption (B5)) are needed to control $\sup_{\mathbb{R}}|\widetilde{f}_{s}-\mathbb{E}[\widetilde{f}_{s}]|$ (respectively $\sup_{\mathbb{R}}|\widetilde{f_{s}}^{\prime}-\mathbb{E}[\widetilde{f}_{s}^{\prime}]|$) thanks to the result of Giné and Guillou (2002). Assumption (B6) ensures that $\sup_{\mathbb{R}}|\widetilde{f_{s}}^{\prime}-f_{s}^{\prime}|$ tends to [math] almost surely as $n\rightarrow\infty$. Assumptions (B3)-(B5) are fulfilled, for instance, by the biweight kernel defined for all $t\in\mathbb{R}$ by:

[TABLE]

Theorem 2**.**

(Asymptotic validity) Let $\varepsilon\in(0,1)$ and let $\mathbb{P}^{*}(.)$ denote the conditional probability given the original data $\mathcal{D}_{n}$. Suppose that assumptions (A3)-(A5), (A7) and (B1)-(B6) are fulfilled. Then, the distribution estimate $\mathbb{P}^{*}(\sup_{\alpha\in[\varepsilon,1-\varepsilon]}|r^{*}_{n}(\alpha)|\leq\cdot)$ given by Algorithm 1 is such that we almost surely have

[TABLE]

where $w_{n}=\sqrt{\log(h_{n}^{-1})/nh_{n}}+h_{n}^{2}$.

The proof is given in Appendix A. Note that under assumption (B2) $w_{n}$ tends to [math] as $n\rightarrow+\infty$. The primary tuning parameters of Algorithm 1 concern the bandwidth $h_{n}$. The optimal bandwidth is obtained by minimizing the term $w_{n}$ with respect to $h_{n}$. This leads to $h_{n}\sim(\log n/n)^{1/5}$ and an approximation error of order $O((\log n/n)^{2/5})$ for the bootstrap estimate (see (Stute, 1982) for details on the derivation of the optimal bandwidth). Notice that assumptions (B2) and (B6) are fulfilled by such a bandwidth.

Although the rate of the bootstrap estimate is slower than that of the Gaussian approximation, the smoothed bootstrap method remains very appealing from a computational perspective: it is indeed very difficult to build confidence bands from simulated Brownian bridges in practice. Finally, as said above, it should be noticed that a non smoothed bootstrap of the ${\rm MV}$ curve would lead to worse approximation rates, of the order $O_{\mathbb{P}}(n^{-1/4})$ namely, see (Falk and Reiss, 1989).

Remark 8**.**

In the result of Theorem 2 we consider the supremum over $\alpha\in[\varepsilon,1-\varepsilon]$ instead of the supremum over $\alpha\in[0,1-\varepsilon]$ for several reasons. One of the reasons is that to obtain a rate of convergence for the bootstrap approximation we need to have the boundedness of the density of the supremum of the absolute value of the Gaussian process $Z_{1}$. This is almost immediate from the result of Pitt and Tran (1979) if the supremum is over $[\varepsilon,1-\varepsilon]$ (see the proof of Theorem 2 in Appendix A for more details). This is more complicated if the supremum is over $[0,1-\varepsilon]$. Using the result of Lifshits (1987), a closer look at the Gaussian process $Z_{1}$ might lead to the boundedness of the density of the supremum of $|Z_{1}|$ over $[0,1-\varepsilon]$. Another reason is that several arguments in the proof also use the fact that $\inf_{[\varepsilon,1-\varepsilon]}f_{s}\circ\alpha_{s}^{-1}>0$. However we do not have $\inf_{[0,1-\varepsilon]}f_{s}\circ\alpha_{s}^{-1}>0$ as under the assumptions of Theorem 2 $f_{s}$ is continuous on $\mathbb{R}$ and therefore $f_{s}(\alpha_{s}^{-1}(0))=0$.

4.2.2 Illustrative Numerical Experiments

Let $x\in\mathbb{R}\mapsto\mathcal{N}(\mu,\Sigma)(x)\in\mathbb{R}$ denote the density of a Gaussian distribution with mean $\mu$ and covariance $\Sigma$. We consider a two-dimensional Gaussian mixture whose density is given by:

[TABLE]

where $\mu_{1}=(0,0)$, $\mu_{2}=(-1,-1)$,

[TABLE]

Density level sets of such a distribution are shown in Fig. 3(a). We draw a sample of size $n=500$ from this two-dimensional Gaussian mixture and compute $\widehat{\rm MV}_{f}$. We then apply Algorithm 1 using the biweight kernel defined in (9) with a bandwidth $h=0.005$ to obtain a $90$% confidence band. We take $\varepsilon=0.05$ and $N=n$ bootstrap replications to approximate $\mathbb{P}^{*}(\sup_{\alpha\in[\varepsilon,1-\varepsilon]}|r_{n}^{*}(\alpha)|\leq\cdot)$. All the volumes required to compute the ${\rm MV}$ curves are estimated using Monte-Carlo integration, drawing uniformly $1,\!000,\!000$ samples in the hypercube enclosing the data. Fig. 2 shows the $90$% confidence curves.

In this section, statistical estimation of the true ${\rm MV}$ curve of a given scoring function $s$ has been investigated, as well as the problem of building confidence regions in the Mass Volume space. In all the rest of the paper, focus is on statistical learning of a nearly optimal scoring function $s$ based on a training sample $X_{1},\dots,X_{n}$ with respect to the ${\rm MV}$ curve criterion.

5 A M-estimation Approach to Anomaly Scoring

Now we are are equipped with the concept of Mass Volume curve, the anomaly scoring task can be formulated as the building of a scoring function $s$, based on the training set $X_{1},\ldots,X_{n}$, such that ${\rm MV}_{s}$ is as close as possible to the optimum ${\rm MV}^{*}$. Due to the functional nature of the criterion performance, there are many ways of measuring how close the ${\rm MV}$ curve of a scoring function candidate and the optimal one are. The $L_{p}$-distances, for $1\leq p\leq+\infty$, provide a relevant collection of risk measures. Let $\varepsilon\in(0,1)$ be fixed (take $\varepsilon=0$ if $\lambda(\mathop{\rm supp}(f))<+\infty$) and consider the losses related to the $L_{1}$-distance and that related to the sup norm:

[TABLE]

Observe that, by virtue of Proposition 1, the ‘excess-risk’ decomposition applies in the $L_{1}$ case and the learning problem can be directly tackled through standard $M$-estimation arguments:

[TABLE]

Hence, possible learning techniques could be based on the minimization, over a set ${\cal S}_{0}\subset{\cal S}$ of candidates, of empirical counterparts of the area under the ${\rm MV}$ curve, such as $\int_{0}^{1-\varepsilon}\widehat{{\rm MV}}_{s}(\alpha)d\alpha$. In contrast, the approach cannot be straightforwardly extended to the sup norm situation. A possible strategy is to combine $M$-estimation with approximation methods so as to ‘discretize’ the functional optimization task. This strategy can be implemented as follows. First, we replace the unknown target curve ${\rm MV}^{*}$ by an approximation that can be described by a finite number of scalar parameters, by a piecewise constant approximant ${\rm MV}^{*}_{\sigma}$ whose breakpoints are given by a subdivision $\sigma:\;0<\alpha_{1}<\dots<\alpha_{K}=1-\varepsilon$ precisely, and that is itself a ${\rm MV}$ curve, namely the ${\rm MV}$ curve of the piecewise constant scoring function

[TABLE]

which is indeed a piecewise constant approximant of ${\rm MV}^{*}$ related to the meshgrid $\sigma$ (see (12)). Then, the $L_{\infty}$-risk can be decomposed as the sum of two terms

[TABLE]

the second term on the right-hand side being viewed as the bias of the statistical method. Restricting optimization to the first term on the right-hand side of the $L_{\infty}$-risk decomposition, the problem thus boils down to recovering the bilevel sets $\mathcal{R}^{*}_{k}=\Omega^{*}_{\alpha_{k}}\setminus\Omega^{*}_{\alpha_{k-1}}$ for $k=1,\ldots,K$ as we obviously have

[TABLE]

This simple observation paves the way for designing scoring strategies relying on the estimation of a finite number of minimum volume sets.

In the next section, we describe a learning algorithm for anomaly ranking, that can be viewed to a certain extent as a statistical version of an adaptive approximation method by piecewise constants introduced in (DeVore, 1987) (see also Section 3.3 in (DeVore, 1998)), to build a piecewise constant estimate of the optimal curve ${\rm MV}^{*}$ and a nearly optimal piecewise constant scoring function, mimicking $s^{*}_{\sigma}$. The subdivision $\sigma$ is entirely learnt from the data, in order to produce accurate estimates of ${\rm MV}^{*}$ in an adaptive fashion: looking at Fig. 1, an ideal meshgrid should be loose where ${\rm MV}^{*}$ is nearly flat or grows very slowly (‘near’ [math]) and refined when it exhibits high degrees of variability (as one gets closer to $1$).

Before describing and analyzing a prototypal approach to ${\rm MV}$ curve optimization, a few remarks are in order.

Remark 9**.**

(Connections with supervised ranking)* Based on the observation made in Remark 3, one may see that, in the specific case where the support of $F(dx)$ coincides with the unit square $[0,1]^{d}$, the ${\rm MV}$ curve of any scoring function $s(x)$ corresponds to the reflection about the first diagonal of the ${\rm ROC}$ curve of $s(x)$ when the ‘negative distribution’ is the uniform distribution on $[0,1]^{d}$ and the ‘positive distribution’ is $F(dx)$. As shown in (Clémençon and Robbiano, 2014), this permits to turn unsupervised ranking into supervised ranking in the compact support situation and to exploit supervised ranking algorithms combined with random sampling to solve the ${\rm MV}$ curve minimization problem. A similar idea had been proposed in (Steinwart et al., 2005) to turn anomaly detection into supervised binary classification.*

Remark 10**.**

(Plug-in)* As the density $f$ is an optimal scoring function, a natural strategy would be to estimate first the unknown density function $f$ by means of (non-) parametric techniques and next use the resulting estimator as a scoring function. Beyond the computational difficulties one would be confronted to for large or even moderate values of the dimension, we point out that the goal pursued in this paper is by nature very different from density estimation: the local properties of the density function are useless here, only the ordering of the possible observations $x\in{\cal X}$ it induces is of importance (see Proposition 1). One may also show that a candidate $f_{1}$ can be a better approximation of the density $f$ than a candidate $f_{2}$ for the $L_{q}$ loss say, but a worse approximation for the ${\rm MV}$ curve criterion (see Example 3).*

Example 3**.**

Let $f$ be the density of the truncated normal distribution with support $[0,1]$, mean equal to 0.5 and variance equal to $0.0225$. Let’s consider $f_{1}:x\in\mathbb{R}\mapsto f(x-0.05)\in\mathbb{R}$ and $f_{2}$ defined for all $x\in\mathbb{R}$ as

[TABLE]

We thus have $\|f_{2}-f\|_{L_{2}}=\|f_{1}-f\|_{L_{2}}+2>\|f_{1}-f\|_{L_{2}}$. Hence $f_{1}$ is a better approximation of $f$ with respect to the $L_{2}$ distance. However ${\rm MV}_{f_{2}}<{\rm MV}_{f_{1}}$ and $f_{2}$ is a better scoring function than $f_{1}$ with respect to the ${\rm MV}$ curve criterion. Indeed one can first show that ${\rm MV}_{f_{2}}={\rm MV}_{f}$. This thus gives ${\rm MV}_{f_{2}}={\rm MV}^{*}\leq{\rm MV}_{f_{1}}$. Now let $\alpha\in(0,1)$, $\Omega_{f_{1},Q(f_{1},\alpha)}=\{x,f_{1}(x)\geq Q(f_{1},\alpha)\}$ is of the form $[0.05-x_{\alpha},0.05+x_{\alpha}]$ with $x_{\alpha}>0$ and $F(\Omega_{s,Q(f_{1},\alpha)})\geq\alpha$. The set $\Omega_{f,Q^{*}(\alpha)}$ is of the form $[-x^{*}_{\alpha},x^{*}_{\alpha}]$ with $x^{*}_{\alpha}>0$ and $F(\Omega_{f,Q^{*}(\alpha)})=\alpha$. If ${\rm MV}_{f_{1}}(\alpha)={\rm MV}^{*}(\alpha)$, i.e., $\lambda(\Omega_{f_{1},Q(f_{1},\alpha)})=\lambda(\Omega_{f,Q^{*}(\alpha)})$, then on the one hand $x_{\alpha}=x^{*}_{\alpha}$ and on the other hand $\Omega_{f_{1},Q(f_{1},\alpha)}=\Omega_{f,Q^{*}(\alpha)}$ $\lambda$ almost everywhere by the uniqueness of the solution of $\eqref{eq:mvpb}$ which is impossible. Eventually, ${\rm MV}_{f_{2}}<{\rm MV}_{f_{1}}$. One can also observe that in contrary to $f_{1}$, $f_{2}$ preserves the order induced by $f$.

6 The A-Rank Algorithm

Now that the anomaly scoring problem has been rigorously formulated, we propose a statistical method to solve it and establish learning rates for the sup norm loss.

6.1 Piecewise Constant Scoring Functions

We focus on scoring functions of the simplest form, piecewise constant functions. Let $K\geq 1$ and consider a partition $\mathcal{W}$ of the feature space $\mathcal{X}$ in $K$ pairwise disjoint subsets of finite Lebesgue measure: $\mathcal{C}_{1},\dots,\mathcal{C}_{K}$ and the subset $\mathcal{X}\setminus\cup_{k=1}^{K}\mathcal{C}_{k}$. When $\lambda(\mathop{\rm supp}(f))$ is finite, one may suppose ${\cal X}\setminus\cup_{k=1}^{K}\mathcal{C}_{k}$ of finite Lebesgue measure. Then, define the piecewise constant scoring function given by:

[TABLE]

Its piecewise constant ${\rm MV}$ curve is given by: $\forall\alpha\in[0,F(\cup_{k=1}^{K}\mathcal{C}_{k})),$

[TABLE]

where $\alpha_{0}=0$, $\alpha_{k}=F(\cup_{j=1}^{k}\mathcal{C}_{j})$ and $\lambda_{k}=\lambda(\cup_{j=1}^{k}\mathcal{C}_{j})$ for all $k\in\{1,\dots,K\}$. If $\lambda(\mathop{\rm supp}(f))$ is finite, ${\rm MV}_{s_{\mathcal{W}}}$ can be defined on the whole interval $[0,1]$.

6.2 Adaptive Approximation of the Optimal MV Curve

Using ideas developed in (DeVore, 1987) (see also Section 3.3 in (DeVore, 1998)), we propose to design an adaptive approximation scheme instead of a subdivision fixed in advance. In such a procedure, the subdivision is progressively refined by adding new breakpoints, as further information about the local variation of the target ${\rm MV}^{*}$ is gained: the subdivision will be coarse where the optimal ${\rm MV}^{*}$ is almost flat and fine where it grows rapidly.

We restrict ourselves only to dyadic subdivisions with breakpoints $\alpha_{j,k}=k(1-\varepsilon)/2^{j}$, with $j\in\mathbb{N}$ and $k\in\{0,\dots,2^{j}\}$ and to partitions of the interval $[0,1-\varepsilon]$ produced by recursive dyadic partitioning: any dyadic subinterval $I_{j,k}=[\alpha_{j,k},\alpha_{j,k+1})$ is possibly split into two halves, producing two siblings $I_{j+1,2k}$ and $I_{j+1,2k+1}$, depending on the local properties of the target ${\rm MV}^{*}$. This adaptive estimation algorithm can be viewed as a top-down search strategy through a binary tree structure.

A binary tree ${\cal T}$ is a tree where all internal nodes have exactly two children. The root of the tree ${\cal T}$ represents the interval $[0,1-\varepsilon]$ and each node of depth $j$ a subinterval $I_{j,k}$. The two children resulting of a split of the node $I_{j,k}$ are the nodes $I_{j+1,2k}$ and $I_{j+1,2k+1}$. A terminal node or a leaf is a node without children. In the sequel, $I_{j,k}$ will denote interchangeably the interval and the related node. Equipped with this flexible tree structure, ${\cal T}$ define a possibly very heterogeneous subdivision $\sigma_{{\cal T}}$ of $[0,1-\varepsilon]$

The approximant associated with the the binary tree ${\cal T}$ is given by: $\forall\alpha\in[0,1-\varepsilon]$,

[TABLE]

where the sum is over all the $j,k$ such that $I_{j,k}$ is a leaf. In order to decide whether a subinterval $I=[\alpha_{1},\alpha_{2}]\subset[0,1-\varepsilon]$ should be split or not, we use the local error on the subinterval $I$ defined by

[TABLE]

The local error $\mathcal{E}(I)$ provides a simple way of estimating the variability of the nondecreasing function ${\rm MV}^{*}$ on the interval $I$. This measure is nonnegative and additive: for any siblings $I_{1}$ and $I_{2}$ of the same subinterval $I$,

[TABLE]

From (DeVore, 1987), we know that $\mathcal{E}$ controls the approximation rate of ${\rm MV}^{*}$ by a constant on any interval $I$ in the sense that:

[TABLE]

where for any function $g:\mathbb{R}\rightarrow\mathbb{R}$, $\|g\|_{I}$ denotes the sup norm over the interval $I$.

Given a tolerance $\tau>0$, the general concept of the algorithm generating a binary tree $\mathcal{T}$ and leading to its associated approximant is as follows: if $\mathcal{E}(I)>\tau$, the interval $I$ is split in two siblings, otherwise $I$ is not split and becomes a leaf. If the algorithm ends we thus have:

[TABLE]

An illustration is given in Fig. 3: Fig. 3(a) shows the piecewise constant approximant of ${\rm MV}^{*}$ as well as the corresponding piecewise scoring function $s_{{\cal T}}(x)$ and Fig. 3(b) shows the related binary tree.

6.3 Empirical Adaptive Estimation of the Optimal MV Curve

The adaptive approximation procedure described above can be turned itself into an adaptive estimation technique. As the optimal curve ${\rm MV}^{*}$ is unknown, the local error $\mathcal{E}(I)$ on a subinterval $I$ is estimated by its empirical counterpart

[TABLE]

where $\widehat{\Omega}_{\alpha}$ is an estimation of the minimum volume set $\Omega^{*}_{\alpha}$. Let $\mathcal{G}$ be a class of measurable subsets of the feature space ${\cal X}$ of finite Lebesgue measure and $\alpha\in(0,1)$. The empirical minimum volume set $\widehat{\Omega}_{\alpha}$ related to the class $\mathcal{G}$ and level $\alpha$ is the solution of the optimization problem:

[TABLE]

where $\phi$ is a penalty term related to the complexity of the class, referred to as the penalty parameter. The accuracy of the solution depends on the choices for the class $\mathcal{G}$ and for the penalty parameter (see (Scott and Nowak, 2006)). In this paper, we do not address the issue of designing algorithms for empirical minimum volume set estimation, we refer to Section 7 of (Scott and Nowak, 2006) for a description of off-the-shelf methods documented in the literature, including partition-based techniques.

As $\mathcal{E}(I)$, the empirical local error $\widehat{\mathcal{E}}(I)$ is nonnegative and additive:

[TABLE]

for any siblings $I_{1}$ and $I_{2}$ of the same subinterval.

Splitting criterion.

The empirical local error $\widehat{\mathcal{E}}$ will serve as a splitting criterion. Given a tolerance $\tau>0$, if $\widehat{\mathcal{E}}(I)>\tau$, the interval $I$ is split in two subintervals $I_{1}$ and $I_{2}$. If $\widehat{\mathcal{E}}(I)\leq\tau$, the interval $I$ is not split and is a leaf of the tree ${\cal T}$.

Stopping criterion.

One can observe that the estimation algorithm does not necessarily terminate for any given tolerance $\tau>0$. Indeed, as $\widehat{F}(\Omega)\in\{k/n:k=0,\dots,n\}$ for any $\Omega\in\mathcal{G}$, the function $\widehat{\lambda}:\alpha\in[0,1)\mapsto\lambda(\widehat{\Omega}_{\alpha})\in\mathbb{R}$ is a piecewise constant function with breakpoints $\{k/n+\phi:k=0,\dots,n-1\}$. This follows by observing that, for any $\alpha\in(k/n+\phi,(k+1)/n+\phi]$, $0\leq k\leq n-1$, the solution of the empirical optimization problem (13) is $\widehat{\Omega}_{(k+1)/n+\phi}$. Therefore if $\tau$ is strictly lower than the minimum of the amplitude of the jumps of the piecewise constant function $\widehat{\lambda}$, the algorithm does not stop.

However, the fact that the function $\widehat{\lambda}$ is a piecewise constant function tells us that the estimation algorithm should stop when the level $j_{\text{max}}=\lfloor\log n/\log 2\rfloor+1$ is reached, where $\lfloor\cdot\rfloor$ denotes the floor part function. Indeed if $j=j_{\text{max}}$, the interval $I_{j,k}=[\alpha_{1},\alpha_{2}]$ is such that $\alpha_{2}-\alpha_{1}<1/n$. This either means that $\alpha_{1}$ and $\alpha_{2}$ belong to the same interval $(k/n+\phi,(k+1)/n+\phi]$ or that $\alpha_{1}\in(k/n+\phi,(k+1)/n+\phi]$ and $\alpha_{2}\in((k+1)/n+\phi,(k+2)/n+\phi]$. In the former case $\lambda(\widehat{\Omega}_{\alpha_{2}})=\lambda(\widehat{\Omega}_{\alpha_{1}})$ and $\widehat{\mathcal{E}}(I_{j,k})=0\leq\tau$. In the latter case, $\widehat{\mathcal{E}}(I_{j,k})$ is equal to the amplitude of the jump of $\widehat{\lambda}$ at $(k+1)/n+\phi$. If this amplitude is lower than $\tau$, $I_{j,k}$ is a leaf. If this amplitude is strictly greater than $\tau$, we should normally split $I_{j,k}$. However this split will not improve the error $\widehat{\mathcal{E}}$ as $\widehat{\mathcal{E}}$ cannot be lower than the amplitude of the jump of $\widehat{\lambda}$. We therefore add a condition $j<j_{\text{max}}$ to the algorithm. This ensures that the algorithm terminates while not changing the output when $\tau$ is greater that the maximum of the amplitudes of the jumps of $\widehat{\lambda}$. Note however that we do not necessarily have $\widehat{\mathcal{E}}(I_{j,k})\leq\tau$ for all $j,k$ such that $I_{j,k}$ is a leaf.

The adaptive estimation method is summarized in Algorithm 2. We denote by $\hat{\sigma}_{\tau}:0=\alpha_{0}<\dots<\alpha_{\widehat{K}}=1-\varepsilon$ the subdivision output by the algorithm such that the empirical piecewise constant estimator of ${\rm MV}^{*}$ (and of ${\rm MV}^{*}_{\hat{\sigma}_{\tau}}$) is given by

[TABLE]

Its bias incorporates in particular that caused by the approximation/discretization stage inherent to the method.

6.4 The Anomaly Ranking algorithm A-Rank

The goal pursued is actually to build a scoring function $\hat{s}(x)$ whose ${\rm MV}$ curve is asymptotically close to the empirical estimate $\widehat{{\rm MV}^{*}}$. In this respect, one should pay attention to the fact that $\widehat{{\rm MV}^{*}}$ is not a ${\rm MV}$ curve in general. Indeed, the sequence of estimated minimum volume sets $\widehat{\Omega}_{\alpha_{k}},k\in\{0,\dots,\widehat{K}\}$ sorted by increasing order of their mass $\alpha_{k}$, is not necessarily increasing (for the inclusion), in contrast to the true minimum level sets $\Omega^{*}_{\alpha_{k}}$. This explains the monotonicity step of the following algorithm.

The A-Rank algorithm is implemented in two stages. Stage 1 consists in running Algorithm 2 to find the breakpoints $\alpha_{0}=0<\alpha_{1}<\dots<\alpha_{\widehat{K}}=1-\varepsilon<1$ and the corresponding empirical minimum volume sets $\widehat{\Omega}_{\alpha_{0}},\dots,\widehat{\Omega}_{\alpha_{\widehat{K}}}$. Stage 2 consists in the monotonicity step before overlaying the estimated sets. The statistical learning method is described at length in Algorithm 3.

Before analyzing the statistical performance of the A-rank algorithm, a few remarks are in order.

Notice first that we could alternatively build a monotone sequence of subsets $(\widetilde{\Omega}_{k})_{0\leq k\leq\widehat{K}}$, recursively through: $\widetilde{\Omega}_{\widehat{K}}=\widehat{\Omega}_{\widehat{K}}$ and $\widetilde{\Omega}_{k}=\widehat{\Omega}_{k}\cap\widetilde{\Omega}_{k+1}$ for $k\in\{\widehat{K}-1,\ldots,0\}$. The results established in the sequel straightforwardly extend to this construction.

One obtains as a byproduct of the algorithm a rough piecewise constant estimator of the (optimal scoring) function $(F_{f}\circ f)(x)$, namely $\sum_{k=1}^{K}(\alpha_{k+1}-\alpha_{k})\cdot\mathbb{I}\{x\in\widetilde{\Omega}_{k}\}$, the Riemann sum approximating the integral (4) when $\mu$ is the Lebesgue measure.

6.5 Performance Bounds for the A-Rank Algorithm

We now prove a result describing the performance of the scoring function produced by the algorithm proposed in the previous section to solve the anomaly scoring problem, as formulated in Section 5. The following assumptions are required.

1. (C1)

$\forall\alpha\in[0,1-\varepsilon]$, $\Omega^{*}_{\alpha}\in\mathcal{G}$. 2. (C2)

Let $(\varepsilon_{i})_{i\geq 1}$ be a Rademacher chaos independent from the $X_{i}$’s. The Rademacher average given by

[TABLE]

where the expectation is over all the random variables, is such that for all $n\geq 1$, $\mathcal{R}_{n}\leq Cn^{-1/2}$, where $C>0$ is a constant.

Assumption (C2) is very general, it is satisfied in particular when the class $\mathcal{G}$ is of finite VC dimension, see (Koltchinskii, 2006). We recall here that for $\delta\in(0,1)$ we have (Scott and Nowak, 2006):

[TABLE]

with $\forall n\geq 1$,

[TABLE]

Therefore assumption (C2) says that the rate of uniform convergence of true to empirical probabilities is of the order $O_{\mathbb{P}}(n^{-1/2})$. Assumption (C1) is in contrast very restrictive, it could be however relaxed at the price of a much more technical analysis, involving the study of the bias in empirical minimum volume set estimation under adequate assumptions on the smoothness of the boundaries of the $\Omega^{*}_{\alpha}$’s, like in Tsybakov (1997) for instance (see Remark 11 below).

We state a rate of convergence for Algorithm 2. The following assumption shall be required.

1. (C3)

The function ${\rm MV}^{*}$ is differentiable on $[0,1)$ with derivative ${\rm MV}^{*\prime}(\alpha)=1/Q^{*}(\alpha)$ for all $\alpha\in[0,1)$. Futhermore the derivative ${\rm MV}^{*\prime}$ belongs to the space $L\log L$ of Borel functions $g:(0,1)\rightarrow\mathbb{R}$ such that:

[TABLE]

As explained in (DeVore, 1987), the space $L\log L$ contains all spaces $L_{p}$, $p>1$ but is strictly contained in $L_{1}$.

Theorem 3**.**

Let $\delta\in(0,1)$. Suppose that assumptions (A1),(A2) and (C1)-(C3) hold true. Take $\tau=5\phi_{n}(\delta)/Q^{*}(1-\varepsilon)$ with $\phi_{n}(\delta)$ as in (16). Then we have with probability at least $1-\delta$:

[TABLE]

In addition, there exists a constant $C>0$ such that with probability at least $1-\delta$ the number of terminal nodes $\text{\emph{card}}(\hat{\sigma}_{\tau})$ output by Algorithm 2 is such that

[TABLE]

The proof of this result can be found in Appendix A. Its argument essentially combines the analysis of the approximation scheme described in Section 6.2 and the generalization bounds for empirical minimum volume sets established in in (Scott and Nowak, 2006), see Theorem 3 and Lemma 19 therein. Here, the rate obtained is not proved to be optimal in the minimax sense, lower bounds will be investigated in a future work.

Corollary 1**.**

Suppose that assumptions of Theorem 3 are satisfied. Given assumption (C2), we have with probability at least $1-\delta$, $\forall n\geq 1$,

[TABLE]

To derive the rate of convergence of the ${\rm MV}$ curve of the estimated scoring function $\hat{s}$ towards ${\rm MV}^{*}$ we need the following assumption on the density $f$:

1. (C4)

There exist constants $\gamma\geq 0$ and $C>0$ such that for all $t>0$ sufficiently small

[TABLE]

This condition (stated with $\mathbb{P}$ instead of $\lambda$) was first introduced in (Polonik, 1995) and used in (Polonik, 1997) to derive rates of convergence of $F(\widehat{\Omega}_{\alpha}\Delta\Omega^{*}_{\alpha})$. It is also used in (Rigollet and Vert, 2009) to obtain fast rates for plug-in estimators of density level sets. The exponent $\gamma$ controls the slope of the function $f$ around any level $q>0$. It is related to Tsybakov’s margin assumption stated for the binary classification framework. The relation between Tsybakov’s margin assumption and the $\gamma$-exponent assumption is given in (Steinwart et al., 2005).

Theorem 4**.**

Let $\delta\in(0,1)$. Suppose that assumptions (A1),(A2) and (C1)-(C4) hold true. Then, with probability at least $1-\delta$,

[TABLE]

Notice that the convergence rate is always slower than $n^{-1/4}$. Furthermore, because of assumption (C1), the convergence rate only depends on the dimension through the parameter $\gamma$ (refer to (Polonik, 1995) for examples of this parameter). However, assumption (C1), made here for simplicity’s sake, is very restrictive and relaxing this assumption is discussed in Remark 11.

Remark 11**.**

(Bias analysis)* Whereas the bias resulting from the discretization of the anomaly ranking problem discussed in Section 6.2 is taken into account in the present rate bound analysis, that related to the approximation of the minimum volume sets $\Omega^{*}_{\alpha}$ has been neglected here for simplicity’s sake (assumption (C1)). We point out that, under smoothness assumptions on the density sublevel sets, it is possible to control such a bias term. Indeed, consider for instance the case where the boundary $\partial\Omega_{\alpha}^{*}$ is of finite perimeter $per(\partial\Omega^{*}_{\alpha})<\infty$ (note that this is the case as soon as $f(x)$ is of bounded variation, the boundary being then $\partial\Omega^{*}_{\alpha}=\{x\in{\cal X}:\;f(x)=t^{*}_{\alpha}\}$ by virtue of $f$’s continuity). In this case, if $\mathcal{G}=\mathcal{G}_{j}$ is the collection of all subsets of $\mathbb{R}^{d}$ obtained by binding together an arbitrary number of hypercubes of side length $2^{-j}$, cartesian products of intervals of the form $[k/2^{j},\;(k+1)/2^{j})$ for $k\in\mathbb{Z}$, the bias term inherent to the estimation of the minimum volume set $\Omega^{*}_{\alpha}$ is bounded by $per(\partial\Omega_{\alpha}^{*})2^{-jd}$, up to a multiplicative constant (see Proposition 9.7 in (Mallat, 1990)). Smaller bounds can be naturally established under more restrictive assumptions involving a regularity parameter $\theta$ of $\partial\Omega^{*}_{\alpha}$, such as its box dimension. Although the optimal choice for $j$ would then depend on $\theta$, a standard fashion of nearly achieving the optimal rate of convergence is to perform model selection (see Section 4 in (Scott and Nowak, 2006)).*

The general approach described above can be extended in various ways. The class $\mathcal{G}$ over which minimum volume set estimation is performed (and the penalty parameter as well) could vary depending on the mass target $\alpha$. Additionally assumption (C1) could be relaxed as discussed in Remark 11. In such a case, if $\Omega^{\mathcal{G}}_{\alpha}$ denotes the solution of the minimum volume set optimization problem (1) over the class $\mathcal{G}$, the error term $\lambda(\widehat{\Omega}_{\alpha})-\lambda(\Omega^{*}_{\alpha})$ could be decomposed into the sum of a stochastic error and an approximation error:

[TABLE]

The choice of the class $\mathcal{G}$ should then balanced the two errors. On the one hand, if the class $\mathcal{G}$ is small, e.g. of finite VC dimension, then the stochastic error can be controlled thanks to assumption (C2) but the approximation error may be very large for some distributions. On the other hand, if the class $\mathcal{G}$ is large, e.g. of infinite VC dimension, then the stochastic error will be very large for some distributions. A model selection approach could thus be incorporated, so as to select an adequate class $\mathcal{G}$ (refer to Section 4 in (Scott and Nowak, 2006)).

In the present analysis, we have restricted our attention to a truncated part of the ${\rm MV}$ space, corresponding to mass levels in the interval $[0,1-\varepsilon]$ and avoiding thus out-of-sample tail regions (in the case where $\mathop{\rm supp}(f)$ is of infinite Lebesgue measure). An interesting direction for further research could consist in investigating the accuracy of anomaly scoring algorithms when letting $\varepsilon=\varepsilon_{n}$ slowly decay to zero as $n\rightarrow+\infty$, under adequate assumptions on the asymptotic behavior of ${\rm MV}^{*}$.

Finally, we emphasize that the A-Rank algorithm is prototypal, the essential objective of its description/analysis in this paper is to provide an insight into the nature of the anomaly scoring issue, viewed here as a continuum of minimum volume set estimation problems. Beyond the possible improvements mentioned above (e.g. target mass level dependent classes $\mathcal{G}$), the main limitation for the practical implementation of such an approach arises from the apparent lack of minimum volume set estimation techniques that can be scaled to high-dimensional settings documented in the literature (except the dyadic recursive partitioning method considered in (Scott and Nowak, 2006), see Section 6.3 therein). However, as observed in (Steinwart et al., 2005) (see also (Clémençon and Robbiano, 2014)), supervised methods combined with sampling procedures can be used for this purpose in moderate dimensions. Refer also to Remark 9.

7 Conclusion

Motivated by a wide variety of applications including health monitoring of complex infrastructures or fraud detection for instance, we have formulated the issue of learning how to rank observations in the same order as that induced by the density function, which we called anomaly scoring here. For this problem, much less ambitious than estimation of the local values taken by the density, a functional performance criterion, the Mass Volume curve namely, is proposed. What ${\rm MV}$ curve analysis achieves for unsupervised anomaly detection is quite akin to what ROC curve analysis accomplishes in the supervised setting.

The statistical estimation of ${\rm MV}$ curves has been investigated from an asymptotic perspective and we have provided a strategy, where the feature space is overlaid with a few well-chosen empirical minimum volume sets, to build a scoring function with statistical guarantees in terms of rate of convergence for the $\sup$ norm in the ${\rm MV}$ space.

Acknowledgements

We are grateful to François Portier and Prof. Patrice Bertail for very helpful discussions. We also thank an anomymous reviewer for raising an important question which led to an improvement of Proposition 1.

Appendix A Proofs

A.1 Properties of the MV Curve

To prove Proposition 1 we need the following lemma.

Lemma 1**.**

The support of the density $\mathop{\rm supp}(f)=\{x,f(x)>0\}$ is equal to the set $\mathcal{Z}_{f}=\{x,F_{f}(f(x))>0\}$ up to subsets of null $\lambda$ measure, i.e.

[TABLE]

The proof is deferred to Appendix B.

Proof of Proposition 1.

We start by proving that all elements of $\mathcal{S}^{*}$ have the same ${\rm MV}$ curve equal to ${\rm MV}_{f}$. One can first show that for all $s\in\mathcal{S}$ and all $\alpha\in(0,1)$, $\alpha^{-1}_{s}(\alpha)=F_{s}^{{\dagger}}(1-\alpha)=Q(s,\alpha)$. Let $s^{*}\in\mathcal{S}^{*}$ and let $\alpha\in(0,1)$,

[TABLE]

where the second equality holds because for any $x$ and any $\alpha\in(0,1)$, $s^{*}(x)\geq F^{{\dagger}}_{s^{*}}(1-\alpha)$ if and only if $F_{s^{*}}(s^{*}(x))\geq 1-\alpha$ (see e.g. Property 2.3 in (Embrechts and Hofert, 2013)). Thanks to assertion $(iv)$ in the definition of $\mathcal{S}^{*}$, for $x\in\overline{\mathop{\rm supp}(f)}$, $F_{s^{*}}(s^{*}(x))=0$ and as $1-\alpha>0$, the term of the last line of the previous equation is equal to [math]. Using the notations introduced in the definition of $\mathcal{S}^{*}$, the term $\lambda\{x\in\mathop{\rm supp}(f),F_{s^{*}}(s^{*}(x))\geq 1-\alpha\}$ can be decomposed as follows:

[TABLE]

The first term is lower than $\lambda(\mathop{\rm supp}(f)\cap\overline{\mathcal{Z}})=0$ (thanks to assertion $(i)$ in the definition of $\mathcal{S}^{*}$) and is therefore equal to [math]. We deal with the second term as follows. Let $x\in\mathop{\rm supp}(f)\cap\mathcal{Z}$,

[TABLE]

The second term is lower than $\mathbb{P}(X\in\overline{\mathop{\rm supp}(f)})=0$ and is therefore equal to 0. Thanks again to assertion $(i)$, the first term is equal to

[TABLE]

where we also used assertions $(ii)$ and $(iii)$. As done above for $s^{*}$, this last term can be shown to be equal to $F_{f}(f(x))$. Therefore,

[TABLE]

and as done above for $s^{*}$, this can be shown to be equal to ${\rm MV}_{f}(\alpha)$.

We now show (3) and $\implies$ in (2). Let $s\in\mathcal{S}$, $\alpha\in(0,1)$. We have,

[TABLE]

Now let $s^{*}\in\mathcal{S^{*}}$. As shown above, we have ${\rm MV}_{s^{*}}={\rm MV}_{f}={\rm MV}^{*}$. Furthermore, $F(\Omega_{s,Q(s,\alpha)})=\mathbb{P}(s(X)\geq F_{s}^{{\dagger}}(1-\alpha))=1-\mathbb{P}(s(X)<F_{s}^{{\dagger}}(1-\alpha))$ and one can show that $\mathbb{P}(s(X)<F_{s}^{{\dagger}}(1-\alpha))\leq 1-\alpha$. We therefore have $\mathbb{P}(\Omega_{s,Q(s,\alpha)})\geq\alpha$. As $\Omega^{*}_{\alpha}$ minimizes $\lambda$ over all sets with mass at least $\alpha$, we have ${\rm MV}_{s}(\alpha)-{\rm MV}^{*}(\alpha)\geq 0$.

Finally we prove $\impliedby$ in (2). On the one hand, for $s=f$ we have ${\rm MV}_{s^{*}}\leq{\rm MV}_{f}$. On the other hand, as $f\in\mathcal{S}^{*}$, thanks to $\implies$ in (2) (which has been proven above), for all scoring function $s$, ${\rm MV}_{f}\leq{\rm MV}_{s}$. Therefore ${\rm MV}_{f}\leq{\rm MV}_{s^{*}}$. We thus have ${\rm MV}_{f}={\rm MV}_{s^{*}}$ on $(0,1)$. From now on, we replace $s^{*}$ by $s$ for the sake of clarity. We thus want to prove that ${\rm MV}_{f}={\rm MV}_{s}$ on $(0,1)$ implies $s\in\mathcal{S}^{*}$. The proof is decomposed in three steps.

First step.

We first show that for $\lambda$-almost all $x\in\mathcal{X}$,

[TABLE]

and

[TABLE]

Let $\alpha\in(0,1)$ be fixed. We have $\lambda(\Omega^{*}_{\alpha})=\lambda(\Omega_{s,Q(s,\alpha)})$. Since $\mathbb{P}(X\in\Omega_{s,Q(s,\alpha)})\geq\alpha$, by uniqueness of the solution of the minimum volume set optimization problem up to subsets of null $\lambda$ measure, we have $\lambda(\Omega^{*}_{\alpha}\Delta\Omega_{s,Q(s,\alpha)})=0$ which is equivalent to: for $\lambda$-almost all $x\in\mathcal{X}$

[TABLE]

i.e. there exists $\mathcal{Z}_{\alpha}\subset\mathcal{X}$ such that $\lambda(\overline{\mathcal{Z}_{\alpha}})=0$ and for all $x\in\mathcal{Z}_{\alpha}$, $\mathbb{I}\{x\in\Omega^{*}_{\alpha}\}=\mathbb{I}\{x\in\Omega_{s,Q(s,\alpha)}\}$. Note that $\overline{\mathcal{Z}_{\alpha}}$ depends on $\alpha$ and we cannot consider to take the union over all $\alpha\in(0,1)$ of the $\overline{\mathcal{Z}_{\alpha}}$ because this union would not necessarily be of null $\lambda$ measure (as we have uncountably many $\alpha$). However considering the union over the rationals of $(0,1)$ is sufficient. Indeed, let $\mathcal{Z}_{\mathbb{Q}}$ denote such a union:

[TABLE]

As it is a union over a countable set of $\alpha$, $\lambda(\overline{\mathcal{Z}_{\mathbb{Q}}})=0$ and for all $x\in\mathcal{Z}_{\mathbb{Q}}=\cap_{\alpha\in(0,1)\cap\mathbb{Q}}\mathcal{Z}_{\alpha}$ and all $\alpha\in(0,1)\cap\mathbb{Q}$,

[TABLE]

Let $x\in\mathcal{Z}_{\mathbb{Q}}$ and let $\alpha\in(0,1)\cap\mathbb{R}\setminus\mathbb{Q}$. We know that there exists a decreasing sequence $(\alpha_{m})_{m\geq 0}$ of elements of $(0,1)\cap\mathbb{Q}$ such that $\alpha_{m}$ converges towards $\alpha$ when $m$ tends to infinity. Now, thanks to a property of the quantile function (see e.g. assertion (5) of Property 2.3 in (Embrechts, 2013)), we have, for all $\alpha^{\prime}\in(0,1)$,

[TABLE]

Thus the sequence $(\Omega^{*}_{\alpha_{m}})_{m\geq 0}$ is decreasing (with respect to set inclusion) and

[TABLE]

as one can show that

[TABLE]

Similarly,

[TABLE]

For all $m\geq 0$, $\mathbb{I}\{x\in\Omega^{*}_{\alpha_{m}}\}=\mathbb{I}\{x\in\Omega_{Q(s,\alpha_{m})}\}$ (because $\alpha_{m}\in(0,1)\cap\mathbb{Q}$) and thus the two limits are equal. Hence the first result.

Now this also implies that for $x\in\mathcal{Z}_{\mathbb{Q}}$,

[TABLE]

If $U$ denotes a random variable with uniform distribution on $(0,1)$, the left-hand side can be rewritten as

[TABLE]

Similarly, the right-hand side is equal to $F_{s}(s(x))$ and therefore for all $x\in\mathcal{Z}_{\mathbb{Q}}$, $F_{f}(f(x))=F_{s}(s(x))$.

Second step.

We can now prove assertions $(i)$, $(ii)$, $(iii)$ and $(iv)$. We will first show that for almost all $x\in\mathop{\rm supp}(f)$, there exists $\alpha^{\prime}\in(0,1]$ such that $f(x)=F_{f}^{{\dagger}}(\alpha^{\prime})$.

One may first observe that $\lambda(\overline{\mathcal{Z}_{f}}\cap\mathop{\rm supp}(f))=0$ where $\mathcal{Z}_{f}=\{x,F_{f}(f(x))>0\}$. Indeed, $\mathcal{Z}_{f}\subset\mathop{\rm supp}(f)$ (if $f(x)=0$ then $F_{f}(f(x))=0$) and therefore $\lambda(\overline{\mathcal{Z}_{f}}\cap\mathop{\rm supp}(f))=\lambda(\mathop{\rm supp}(f)\setminus\mathcal{Z}_{f})$ which is equal to 0 since, thanks to Lemma 1, $\lambda(\mathop{\rm supp}(f)\Delta\mathcal{Z}_{f})=0$.

Let $x\in\mathop{\rm supp}(f)\cap\mathcal{Z}_{f}$. If $F_{f}^{{\dagger}}$ is continuous at $\alpha^{\prime}=F_{f}(f(x))\in(0,1]$ then $F_{f}^{{\dagger}}(\alpha^{\prime})=F_{f}^{{\dagger}}(F_{f}(f(x)))=f(x)$. Now, since $F_{f}^{{\dagger}}$ is increasing, $F_{f}^{{\dagger}}$ has at most countably many discontinuities and each discontinuity $m$ corresponds to a jump of $F_{f}^{{\dagger}}$ between two values $t^{m}_{1}$ and $t^{m}_{2}$. We also know that this jump of $F_{f}^{{\dagger}}$ corresponds to a flat part of $F_{f}$ between $t^{m}_{1}$ and $t^{m}_{2}$. However if $F_{f}$ has a flat part between $t^{m}_{1}$ and $t^{m}_{2}$ then $\mathbb{P}(t^{m}_{1}\leq f(X)\leq t^{m}_{2})=F_{f}(t^{m}_{2})-F_{f}(t^{m}_{1})=0$ (recall that $F_{f}$ is continuous as $f$ has no flat parts). This gives

[TABLE]

Thus $f=0$ $\lambda$-almost everywhere on $\{u,t^{m}_{1}\leq f(u)\leq t^{m}_{2}\}$. This implies that $\lambda(\{x,t^{m}_{1}\leq f(x)\leq t^{m}_{2}\}\cap\mathop{\rm supp}(f))=0$. Let $\overline{\mathcal{Z}_{0}}$ be the union of such sets over $m\geq 0$:

[TABLE]

and let $\mathcal{Z}_{1}=\mathcal{Z}_{0}\cap\mathcal{Z}_{f}$. We have $\lambda(\overline{\mathcal{Z}_{1}}\cap\mathop{\rm supp}(f))=0$ and for all $x\in\mathcal{Z}_{1}\cap\mathop{\rm supp}(f)$, there exists $\alpha^{\prime}\in(0,1]$ such that $f(x)=F_{f}^{{\dagger}}(\alpha^{\prime})$ because $F_{f}^{{\dagger}}$ is continuous at $\alpha^{\prime}=F_{f}(f(x))$ (otherwise this would imply that $x$ belongs to a region corresponding to a flat part of $F_{f}$, i.e. one of the sets $\{x,t_{1}^{m}\leq f(x)\leq t_{2}^{m}\}$).

Similarly, we can show that there exists a set $\mathcal{Z}_{2}$ such that $\lambda(\overline{\mathcal{Z}_{2}}\cap\mathop{\rm supp}(f))=0$ and for all $x\in\mathcal{Z}_{2}\cap\mathop{\rm supp}(f)$, there exists $\alpha^{\prime}\in(0,1]$ such that $s(x)=F_{s}^{{\dagger}}(\alpha^{\prime})$. The only difference with $f$ is that we here have to consider $\mathcal{Z}_{\mathbb{Q}}$ so that $F_{s}(s(x))=F_{f}(f(x))$.

Let $\mathcal{Z}=\mathcal{Z}_{\mathbb{Q}}\cap\mathcal{Z}_{1}\cap\mathcal{Z}_{2}$. We have $\lambda(\overline{\mathcal{Z}}\cap\mathop{\rm supp}(f))=0$. Let $x_{1},x_{2}\in\mathcal{Z}\cap\mathop{\rm supp}(f)$ such that $f(x_{1})>f(x_{2})$ and let $\Omega_{1}=\{x\in\mathcal{Z},f(x)\geq f(x_{1})\}$. There exists $\alpha_{1}\in[0,1)$ such that $F_{f}^{{\dagger}}(1-\alpha_{1})=f(x_{1})$. Let’s first consider the case where $\alpha_{1}\in(0,1)$. We have

[TABLE]

As $x_{1}\in\Omega_{1}$, $x_{1}\in\{x\in\mathcal{Z},s(x)\geq F_{s}^{{\dagger}}(1-\alpha_{1})\}$ and $s(x_{1})\geq F_{s}^{{\dagger}}(1-\alpha_{1})$. Analogously, as $x_{2}\notin\Omega_{1}$, $x_{2}\notin\{x\in\mathcal{Z},s(x)\geq F_{s}^{{\dagger}}(1-\alpha_{1})\}$ and $s(x_{2})<F_{s}^{{\dagger}}(1-\alpha_{1})$. Thus $s(x_{1})>s(x_{2})$. Let’s now consider the case $\alpha_{1}=0$. We do not know that (17) still holds. However we can always assume without loss of generality that $s$ is bounded. Indeed, if $s$ is not bounded, consider the function $s^{\prime}=\arctan\circ s$. As $\arctan$ is strictly increasing, thanks to Proposition 3, ${\rm MV}_{s}={\rm MV}_{s^{\prime}}$. Besides, as $s$ takes its values in $\mathbb{R}^{+}$, $s^{\prime}$ takes its values in $[0,\pi/2)$ and is bounded. Finally, if we show that $s^{\prime}=T\circ f$ with $T$ strictly increasing then we also have that $s$ is a strictly increasing transform of $f$. Then as $s$ is bounded, $F_{s}^{{\dagger}}(1-\alpha_{1})$ is defined at $\alpha_{1}=0$ and we can replace ${\rm MV}_{s}={\rm MV}^{*}$ on $(0,1)$ by ${\rm MV}_{s}={\rm MV}^{*}$ on $[0,1)$ in the proposition. We can then show that $\eqref{eq:set_equality}$ holds for $\alpha_{1}=0$.

Similarly, $s(x_{1})>s(x_{2})$ implies $f(x_{1})>f(x_{2})$. Therefore for all $x_{1},x_{2}\in\mathcal{Z}\cap\mathop{\rm supp}(f)$,

[TABLE]

and hence the existence of $T$. One can for instance take $T:f(x)\in f(\mathcal{Z}\cap\mathop{\rm supp}(f))\mapsto s(x)\in s(\mathcal{Z}\cap\mathop{\rm supp}(f))$.

Third step.

Finally, assertion $(iv)$ derives from the fact that for $x\in\mathcal{Z}_{\mathbb{Q}}\cap\overline{\mathop{\rm supp}(f)}$ we have $F_{s}(s(x))=F_{f}(f(x))=0$. ∎

Proof of Proposition 2.

To prove the convexity of ${\rm MV}^{*}$ we show that the slopes are increasing. Let $\alpha_{1}<\alpha_{2}<\alpha_{3}$ be in $[0,1)$,

[TABLE]

where the third equality holds because for any $x$ and any $\alpha\in[0,1)$, $f(x)\geq F^{{\dagger}}_{f}(1-\alpha)$ if and only if $F_{f}(f(x))\geq 1-\alpha$ (see e.g. Property 2.3 in (Embrechts and Hofert, 2013)). Now as $x\in\{x,F_{f}^{{\dagger}}(1-\alpha_{2})\leq f(x)<F_{f}^{{\dagger}}(1-\alpha_{1})\}$, we obtain

[TABLE]

because the distribution of $F_{f}(f(X))$ is the uniform distribution on $(0,1)$ (as $F_{f}$ is continuous since $f$ has no flat parts). Similarly, one can show that ${\rm MV}^{*}(\alpha_{3})-{\rm MV}^{*}(\alpha_{2})\geq(\alpha_{3}-\alpha_{2})/F_{f}^{{\dagger}}(1-\alpha_{2})$. Hence, we finally have

[TABLE]

We now prove the differentiability of ${\rm MV}^{*}$ and the formula of its derivative. First as $F_{f}$ is invertible on $(a,F_{f}^{{\dagger}}(1)]$, $F_{f}^{{\dagger}}$ is the ordinary inverse of $F_{f}$ on $(0,1]$. Let $\alpha\in(0,1)$ and $h>0$. Proceeding similarly as above we can show that

[TABLE]

Therefore as $F_{f}^{{\dagger}}$ is continuous (because $F_{f}^{{\dagger}}$ is the ordinary inverse of $F_{f}$),

[TABLE]

Analogously, we also have

[TABLE]

which implies that ${\rm MV}^{\prime*}(\alpha)=1/F_{f}^{{\dagger}}(1-\alpha)$. For $\alpha=0$, we have ${\rm MV}^{*}(0)=0$ and proceeding as for $\alpha\in(0,1)$, if $h>0$,

[TABLE]

and

[TABLE]

Therefore $\lim_{h\rightarrow 0^{+}}{\rm MV}^{*}(h)/h=1/F_{f}^{{\dagger}}(1)$. Thus, as $Q^{*}(\alpha)=F_{f}^{{\dagger}}(1-\alpha)$ for all $\alpha\in[0,1)$ we obtain the formula of the derivative of ${\rm MV}^{*}$. ∎

A.2 Statistical Estimation of the MV curve

A.2.1 Strong approximation: proof of Theorem 1

To prove Theorem 1 we need the following lemma.

Lemma 2**.**

Under assumptions (A3)-(A4) and (A6) there exists a constant $C>0$ such that we almost surely have for $n$ large enough,

[TABLE]

Proof of Lemma 2.

Let $U_{i}=F_{s}(s(X_{i}))$ for all $i\in\{1,\dots,n\}$. As $F_{s}$ is continuous, the random variables $U_{1},\dots,U_{n}$ are i.i.d. with uniform distribution on $(0,1)$. Their empirical cumulative distribution function is given by $\widehat{U}(\alpha)=1/n\sum_{i=1}^{n}\mathbb{I}\{U_{i}\leq\alpha\}$ for all $\alpha\in[0,1]$. Furthermore we have for all $\alpha\in(0,1]$, $F_{s}(\widehat{F}_{s}^{{\dagger}}(\alpha))=\widehat{U}^{{\dagger}}(\alpha)$. We also define $\widehat{U}^{{\dagger}}(0)=0$. Let $U_{n}$ denote the uniform empirical process defined for all $\alpha\in[0,1]$ by $U_{n}(\alpha)=\sqrt{n}(\widehat{U}(\alpha)-\alpha)$ and $u_{n}$ denote the uniform empirical quantile process defined for all $\alpha\in[0,1]$ by $u_{n}(\alpha)=\sqrt{n}(\widehat{U}^{{\dagger}}(\alpha)-\alpha)$. Applying the Dvoretzky-Kiefer-Wolfowitz (DKW) inequality (Massart, 1990) gives for all $z>0$ and all $n\geq 1$,

[TABLE]

As $\sup_{0\leq\alpha\leq 1}|U_{n}(\alpha)|=\sup_{0\leq\alpha\leq 1}|u_{n}(\alpha)|$ for all $\omega$ in the sample space (see equation (1.4.5) in (Csörgő, 1983)), the DKW inequality (18) holds when replacing $U_{n}$ by $u_{n}$. Taking $z=\sqrt{\log n}$, we obtain for all $n\geq 1$,

[TABLE]

As $\sum_{n=1}^{\infty}1/n^{2}<\infty$, by the Borel-Cantelli lemma, we almost surely have for $n$ large enough,

[TABLE]

Using the mean value theorem we can write for $\alpha^{\prime}\in[\varepsilon,1]$,

[TABLE]

where $\xi$ is between $\alpha^{\prime}$ and $\widehat{U}^{{\dagger}}(\alpha^{\prime})$. As $\widehat{U}^{{\dagger}}$ is increasing we have for all $\alpha^{\prime}\in[\varepsilon,1]$, $\widehat{U}^{{\dagger}}(\varepsilon)\leq\widehat{U}^{{\dagger}}(\alpha^{\prime})\leq 1$. From (20) we know that there exists a constant $c_{1}>0$ such that we almost surely have for $n$ large enough, $\widehat{U}^{{\dagger}}(\varepsilon)\geq\varepsilon-c_{1}>0$. Therefore, almost surely and for $n$ large enough,

[TABLE]

Note that the last infimum is strictly positive because as $f_{s}\circ\alpha_{s}^{-1}$ is continuous, the infimum is reached at a point $y_{0}\in[0,1-\varepsilon+c_{1}]\subset[0,1)$ and as $f_{s}\circ\alpha_{s}^{-1}>0$ on $[0,1)$ (assumptions (A4) and (A6)), $f_{s}(\alpha_{s}^{-1}(y_{0}))>0$.

This eventually leads to: almost surely and for $n$ large enough,

[TABLE]

∎

Proof of assertion $(i)$ of Theorem 1.

Proof.

Assume that assumptions (A3)-(A4) and (A6)-(A7) are fulfilled. Using the mean value theorem we can write for all $\alpha\in[0,1-\varepsilon]$

[TABLE]

where $\xi$ is between $\widehat{\alpha}_{s}^{-1}(\alpha)$ and $\alpha_{s}^{-1}(\alpha)$. Therefore,

[TABLE]

Now for all $\alpha\in[0,1-\varepsilon]$, $\alpha_{s}^{-1}(\alpha)$ and $\widehat{\alpha}_{s}^{-1}(\alpha)$ belong to the support of $f_{s}$, i.e., $[0,\|s\|_{\infty}]$. This is true for $\widehat{\alpha}_{s}^{-1}(\alpha)$ because it is equal to one of the $s(X_{i})$, $1\leq i\leq n$. Thus $\xi\in[0,\|s\|_{\infty}]$. As $\lambda_{s}^{\prime}$ is continuous, we have

[TABLE]

and

[TABLE]

From Lemma 2 we know that there exists a constant $C>0$ such that we almost surely have for $n$ large enough,

[TABLE]

Therefore $\sup_{\alpha^{\prime}\in[\varepsilon,1]}|\widehat{F}_{s}^{{\dagger}}(\alpha^{\prime})-F_{s}^{{\dagger}}(\alpha^{\prime})|$ converges almost surely to [math] and so is $\sup_{\alpha\in[0,1-\varepsilon]}|\widehat{\rm MV}_{s}(\alpha)-{\rm MV}_{s}(\alpha)|$. ∎

Proof of assertion $(ii)$ of Theorem 1.

Proof.

For all $n\geq 1$, let $v_{n}=n^{-1/2}\log n$. Assume that assumptions (A3)-(A7) are fulfilled. By virtue of Theorem 3.1.2 in (Csörgő, 1983), there exists a sequence of Brownian bridges $\{B_{n}^{1}(\alpha),\alpha\in[0,1]\}_{n\geq 1}$ such that, we almost surely have:

[TABLE]

where $u_{n}$ is the uniform quantile process as defined in the proof of Lemma 2. From (3.3) of Theorem 3 in (Csörgő and Révész, 1978), we also have almost surely, for $n$ large enough,

[TABLE]

where $C_{1}=40c10^{c}$ and $a_{n}=25n^{-1}\log\log n$. Hence for $n$ large enough we have $a_{n}\leq\varepsilon$ and therefore we almost surely have

[TABLE]

Now from the proof of theorem 3.2.1 in (Csörgő, 1983), we also almost surely have,

[TABLE]

Combining (21), (22) and (23), we almost surely have:

[TABLE]

As one can show that $F_{s}^{{\dagger}}(\alpha)=\alpha_{s}^{-1}(1-\alpha)$ and $\widehat{F}_{s}^{{\dagger}}(\alpha)=\widehat{\alpha}_{s}^{-1}(1-\alpha)$ for all $\alpha\in(0,1]$ we almost surely have

[TABLE]

With the change of variable $\alpha=1-\alpha^{\prime}$ this leads to

[TABLE]

where $\{B_{n}(\alpha),\alpha\in[0,1]\}_{n\geq 1}$ is the sequence of stochastic processes defined by $\{B_{n}(\alpha),\alpha\in[0,1]\}_{n\geq 1}=\{B_{n}^{1}(1-\alpha),\alpha\in[0,1]\}_{n\geq 1}$. Notice that for all $n\geq 1$, $B_{n}$ is a Brownian Bridge as it has the same distribution as $B_{n}^{1}$. Thus there exists a constant $C_{2}$ independent of $\alpha$ such that almost surely for $n$ large enough and for all $\alpha\in[0,1-\varepsilon]$,

[TABLE]

Hence, dividing by f_{s}\bigl{(}\alpha_{s}^{-1}(\alpha)\bigr{)} which is strictly positive for all $\alpha\in[0,1-\varepsilon]$ (assumptions (A4) and (A6)), we almost surely have, for $n$ large enough and for all $\alpha\in[0,1-\varepsilon]$,

[TABLE]

where $\inf_{0\leq\alpha\leq 1-\varepsilon}f_{s}(\alpha_{s}^{-1}(\alpha))$ is strictly positive by the same argument as the one given in the proof of Lemma 2. Now using a Taylor expansion of $\lambda_{s}$ which is of class $\mathcal{C}^{2}$ (assumption (A7)) we can write for all $\alpha\in[0,1-\varepsilon]$:

[TABLE]

where $R_{2}(\alpha)=\lambda_{s}^{{}^{\prime\prime}}(\xi)(\widehat{\alpha}_{s}^{-1}(\alpha)-\alpha_{s}^{-1}(\alpha))^{2}/2$ is the remainder of the Taylor expansion, $\xi$ being between $\alpha_{s}^{-1}(\alpha)$ and $\widehat{\alpha}_{s}^{-1}(\alpha)$. We thus have

[TABLE]

For the first term we almost surely have, for all $\alpha\in[0,1-\varepsilon]$ and for $n$ large enough,

[TABLE]

For all $\alpha\in[0,1-\varepsilon]$, $\alpha_{s}^{-1}(\alpha)\in[0,\|s\|_{\infty}]$. As $\lambda_{s}^{\prime}$ is continuous, $|\lambda_{s}^{\prime}(\alpha_{s}^{-1}(\alpha))|$ is bounded by $\sup_{[0,\|s\|_{\infty}]}|\lambda_{s}^{\prime}|$.

We treat the second term as follows. For all $\alpha\in[0,1-\varepsilon]$, $\alpha_{s}^{-1}(\alpha)$ and $\widehat{\alpha}_{s}^{-1}(\alpha)$ are in $[0,\|s\|_{\infty}]$. Therefore $\xi\in[0,\|s\|_{\infty}]$ and as $\lambda_{s}^{\prime\prime}$ is continuous, $|\lambda_{s}^{\prime\prime}(\xi)|$ is bounded by $\sup_{[0,\|s\|_{\infty}]}|\lambda_{s}^{\prime\prime}|$. Thus,

[TABLE]

where the last inequality holds almost surely and for $n$ large enough and where $C>0$ is the constant of Lemma 2.

Eventually, combining the bounds on $\sup_{[0,1-\varepsilon]}|C_{n}|$ and $\sup_{[0,1-\varepsilon]}|D_{n}|$ we almost surely have

[TABLE]

This concludes the proof. ∎

A.2.2 Bootstrap consistency: Proof of Theorem 2

Recall that for all $n\geq 1$, $v_{n}=n^{-1/2}\log n$ and $w_{n}=\sqrt{\log(h_{n}^{-1})/(nh_{n})}+h_{n}^{2}$. Let $y_{s}(\alpha)$ and $\widetilde{y}_{s}(\alpha)$ be respectively defined for all $\alpha\in[\varepsilon,1-\varepsilon]$ as $y_{s}(\alpha)=\lambda_{s}^{\prime}(\alpha_{s}^{-1}(\alpha))/f_{s}(\alpha_{s}^{-1}(\alpha))$ and $\widetilde{y}_{s}(\alpha)=\lambda_{s}^{\prime}(\widetilde{\alpha}_{s}^{-1}(\alpha))/\widetilde{f}_{s}(\widetilde{\alpha}_{s}^{-1}(\alpha))$.

Sketch of proof.

The argument is based on a strong approximation type inequality for $r_{n}$ (see Lemma 3) and $r_{n}^{*}$ (see Lemma 4). These inequalities can then be used to obtain a result on the rate of convergence of the cumulative distribution function of $\sup_{[\varepsilon,1-\varepsilon]}|r_{n}|$ (respectively, the conditional cumulative distribution function of $\sup_{[\varepsilon,1-\varepsilon]}|r_{n}^{*}|$) towards the supremum of a Gaussian process which depends on $y_{s}$ (respectively the smoothed empirical version $\widetilde{y}_{s}$ of $y_{s}$). We finally obtain the result using the rate of the strong uniform convergence of $\widetilde{y}_{s}/y_{s}$ towards $1$ on $[\varepsilon,1-\varepsilon]$ (see Lemma 5). The technical proofs of Lemma 3, Lemma 4 and Lemma 5 are deferred to Appendix B for the sake of clarity of the proof of the main result.

We first need the following strong approximation result.

Lemma 3**.**

Let $\varepsilon\in(0,1)$. There exists a positive constant $C$ such that

[TABLE]

We also need the counterpart of Lemma 3 conditionally on the data set $\mathcal{D}_{n}$.

Lemma 4**.**

Let $\varepsilon\in(0,1)$. Under assumptions (A3)-(A5), (A7) and (B1)-(B6), there exists a constant $C$ such that we $\mathbb{P}$-almost surely have,

[TABLE]

where

[TABLE]

and where for each $n\geq 1$ the distribution of $B_{n}^{*}$ conditionally to $\mathcal{D}_{n}$ is the one of a Brownian bridge.

Eventually, to control the distance between the distribution of $\sup_{[\varepsilon,1-\varepsilon]}|Z_{n}|$ and the conditional distribution of $\sup_{[\varepsilon,1-\varepsilon]}|Z_{n}^{*}|$ we need the following lemma.

Lemma 5**.**

Under assumptions (A3) and (B1)-(B4) there exists a constant $C>0$ such that we almost surely have, for $n$ large enough,

[TABLE]

We can now proof the main result.

Proof of Theorem 2.

Let $\varepsilon\in(0,1)$. Let $C>0$ be the constant of Lemma 3. If $\sup_{\alpha\in[\varepsilon,1-\varepsilon]}|r_{n}(\alpha)-Z_{n}(\alpha)|\leq Cv_{n}$ then

[TABLE]

Thus thanks to the result of Lemma 3

[TABLE]

Therefore using a result of Sargan and Mikhail (1971), we have for all $t\in\mathbb{R}$,

[TABLE]

and

[TABLE]

where $\phi_{\bar{Z}}$ denotes the density of $\sup_{[\varepsilon,1-\varepsilon]}|Z_{1}|$ and where for the first equality we use the fact that for all $n\geq 1$ the random variable $Z_{n}(\alpha)$ is equal in distribution to $Z_{1}(\alpha)$ and for the last inequality the result from (Pitt and Tran, 1979) stating that $\sup_{\alpha\in[\varepsilon,1-\varepsilon]}|Z_{1}(\alpha)|$ has a density bounded by a constant $D>0$ as the supremum of a Gaussian process $\{Z_{1}(\alpha),\varepsilon\leq\alpha\leq 1-\varepsilon\}$ such that for all $\alpha\in[\varepsilon,1-\varepsilon]$, $\inf_{\alpha\in[\varepsilon,1-\varepsilon]}\text{Var}[Z_{1}(\alpha)]>0$ and $\mathbb{P}(\sup_{\alpha\in[\varepsilon,1-\varepsilon]}|Z_{1}(\alpha)|<\infty)=1$. Indeed, as $B_{1}$ is a Brownian bridge it is almost surely continuous on $[0,1]$ and as $y_{s}$ is continuous on $[\varepsilon,1-\varepsilon]$, $Z_{1}$ is almost surely continuous on $[\varepsilon,1-\varepsilon]$ and therefore almost surely $\sup_{\alpha\in[\varepsilon,1-\varepsilon]}|Z_{1}(\alpha)|<\infty$. Furthermore, $\alpha\in[\varepsilon,1-\varepsilon]\mapsto\text{Var}[Z_{1}(\alpha)]=\alpha(1-\alpha)y_{s}^{2}(\alpha)$ is continuous and thanks to Lemma 10, for all $\alpha\in[\varepsilon,1-\varepsilon]$, $\text{Var}[Z_{1}(\alpha)]>0$. Therefore $\inf_{\alpha\in[\varepsilon,1-\varepsilon]}\text{Var}[Z_{1}(\alpha)]$ is attained and is strictly positive.

Therefore,

[TABLE]

Reasoning similarly as above, thanks to the result of Lemma 4, there exists a constant $C_{1}$ such that, $\mathbb{P}$-almost surely, as $n\rightarrow\infty$,

[TABLE]

Therefore using the result of (Sargan and Mikhail, 1971), we $\mathbb{P}$-almost surely have, for all $t\in\mathbb{R}$,

[TABLE]

We now need the following result.

Lemma 6**.**

The exists a constant $C_{2}>0$ such that we $\mathbb{P}$-almost surely have for $n$ large enough,

[TABLE]

Proof of Lemma 6.

Let $t\in\mathbb{R}$. If $t<0$ both probabilities are equal to 0 as the two random variables involved are positive. Hence we have (for all $\omega$ in the sample space)

[TABLE]

Let $t\geq 0$. Thanks to Lemma 5 there exists a constant $C_{3}>0$ independent of $\alpha$ such that we almost surely have for $n$ large enough:

[TABLE]

Furthermore, we can assume, given $n$ large enough, that there exists a constant $C_{4}>0$ such that $1-C_{3}w_{n}>C_{4}$ as $w_{n}$ tends towards [math]. Observe that

[TABLE]

and decompose the last term as follows

[TABLE]

where the event $\mathcal{Z}$ is defined as

[TABLE]

and its complementary is given by

[TABLE]

The first term of (28) is lower than

[TABLE]

where the last equality stands from the fact that the law of $B^{*}_{1}$ conditionally on $\mathcal{D}_{n}$ is equal to the law of $B_{1}$. The second term of (28) is lower than

[TABLE]

which is $\mathbb{P}$-almost surely equal to [math] thanks to (27). Note that the fact that this holds $\mathbb{P}$-almost surely is independent of $t$. Therefore we $\mathbb{P}$-almost surely have: for all $t\geq 0$,

[TABLE]

where we use a Taylor expansion and where $\xi$ is between $t$ and $t/(1-C_{3}w_{n})$.

For $n$ large enough, we have $1>1-C_{3}w_{n}>C_{4}>0$. On the one hand this gives $1<1/(1-C_{3}w_{n})$ and, as $t\geq 0$, $t<t/(1-C_{3}w_{n})$. Therefore, as $\xi$ is between $t$ and $t/(1-C_{3}w_{n})$, we have in fact, $t<\xi<t/(1-C_{3}w_{n})$. Thus $\phi_{\bar{Z}}(\xi)t\leq\phi_{\bar{Z}}(\xi)\xi$. On the other hand, we also have $1/(1-C_{3}w_{n})<1/C_{4}$. This eventually gives $\mathbb{P}$-almost surely, for all $t\geq 0$,

[TABLE]

A lower bound can be obtained in a similar fashion: $\mathbb{P}$-almost surely, for all $t\geq 0$,

[TABLE]

Combining the two inequalities (29) and (30) we obtain that there exists a constant $C_{5}>0$ such that we $\mathbb{P}$-almost surely have

[TABLE]

Let $\varphi$ be the density of the Gaussian distribution $\mathcal{N}(0,1)$. Using a remark in the paper of (Tsirel’son, 1976, p. 854), there exists $M>0$ such that for all $x\geq M$,

[TABLE]

where $v_{1}$ is such that $v_{1}^{2}=\sup_{\alpha\in[\varepsilon,1-\varepsilon]}\text{Var}[|Z_{1}(\alpha)|]$, $a>0$ is arbitrary and $b\geq 0$ only depends on $v_{1}$, $\phi_{\bar{Z}}$ and the cumulative distribution function of the Gaussian distribution $\mathcal{N}(0,1)$. Now the function $x\in\mathbb{R}^{+}\mapsto x/v_{1}\cdot\varphi((x-a-b)/v_{1})$ is bounded on $\mathbb{R}^{+}$ as it is continuous and tends towards [math] when $x$ tends to $+\infty$. Therefore from (31) we have that $\sup_{x\geq M}|x\phi_{\bar{Z}}(x)|<+\infty$. And as from (Pitt and Tran, 1979) $\phi_{\bar{Z}}$ is bounded on $\mathbb{R}$, so is $x\mapsto x\phi_{\bar{Z}}(x)$ on $[0,M]$. Eventually, $\sup_{x\in\mathbb{R}^{+}}|x\phi_{\bar{Z}}(x)|<+\infty$ and there exists a constant $C_{2}>0$ such that $\mathbb{P}$-almost surely,

[TABLE]

Combining (26) and (32) we obtain the result of the Lemma. ∎

From Lemma 6, we $\mathbb{P}$-almost surely have

[TABLE]

and thus

[TABLE]

Finally combining this last inequality with (25) we obtain the result of the theorem.

∎

A.3 A-Rank Algorithm

A.3.1 Proof of Theorem 3

The proof follows closely the one of theorem 2 in (Clémençon and Vayatis, 2009). For clarity, we start off with recalling the following result.

Proposition 4**.**

(Scott and Nowak (2006)) Suppose that the assumptions of Theorem 3 are fulfilled. For any $\alpha\in(0,1)$, let $\widehat{\Omega}_{\alpha}\in\mathcal{G}$ be a solution of the constrained optimization problem:

[TABLE]

Let $\delta\in(0,1)$. With probability at least $1-\delta$: $\forall\alpha\in(0,1)$, $\forall n\geq 1$,

[TABLE]

The major part of this result is given by Theorem 3 and Lemma 19 in (Scott and Nowak, 2006). Although they prove the result for a given $\alpha$, it can be extended to all $\alpha\in(0,1)$. The upper bound on $F(\widehat{\Omega}_{\alpha})$ can be obtained with the following reasoning. Suppose that we have $F(\widehat{\Omega}_{\alpha})>\alpha$, then $F(\widehat{\Omega}_{\alpha})\geq\alpha$ and $\lambda(\Omega^{*}_{\alpha})\leq\lambda(\widehat{\Omega}_{\alpha})$ because $\Omega^{*}_{\alpha}$ minimizes $\lambda$ over all measurable sets $\Omega$ such that $F(\Omega)\geq\alpha$. The upper bound on the volume gives $\lambda(\widehat{\Omega}_{\alpha})\leq\lambda(\Omega^{*}_{\alpha})$, thus $\lambda(\widehat{\Omega}_{\alpha})=\lambda(\Omega^{*}_{\alpha})$. Hence $\widehat{\Omega}$ also minimizes $\lambda$ over all measurable sets $\Omega$ such that $F(\Omega)\geq\alpha$ and by uniqueness of the solution under assumptions (A1) and (A2), we have $F(\widehat{\Omega}_{\alpha})=F(\Omega^{*}_{\alpha})=\alpha$ which contradicts $F(\widehat{\Omega}_{\alpha})>\alpha$.

We now prove the following lemma quantifying the uniform deviation of the empirical local error from the true local error over all dyadic scales.

Lemma 7**.**

Suppose that assumptions (C1) and (C2) are satisfied . Let $\delta\in(0,1)$. With probability at least $1-\delta$, we have: $\forall n\geq 1$,

[TABLE]

Proof.

Let $0\leq j\leq j_{\text{max}}$, $0\leq k\leq 2^{j}-1$,

[TABLE]

From proposition 4, with probability at least $1-\delta$ we have: for all $0\leq j\leq j_{\text{max}},0\leq k\leq 2^{j}$,

[TABLE]

therefore with probability at least $1-\delta$, for all $0\leq j\leq j_{\text{max}}$, $0\leq k\leq 2^{j}-1$,

[TABLE]

and the result follows. ∎

From Lemma 7, with probability at least $1-\delta$, for all $0\leq j\leq j_{\text{max}}$, $0\leq k\leq 2^{j}-1$,

[TABLE]

Let $E_{\sigma_{\tau}}({\rm MV}^{*})$ be the piecewise constant approximant of the optimal ${\rm MV}$ curve built from the same recursive strategy as the one implemented in Algorithm 2 but based on the true local error $\mathcal{E}$ and without the condition $j<j_{\text{max}}$, denoting $\sigma_{\tau}$ the associated meshgrid. We thus have, for every $\alpha\in[0,1-\varepsilon]$:

[TABLE]

We also use the notation $E_{\widehat{\sigma}_{\tau}}$ to denote the function defined for all $\alpha\in[0,1-\varepsilon]$ as

[TABLE]

where we recall that $\widehat{\sigma}_{\tau}$ is the meshgrid obtained in Algorithm 2 implemented with the empirical error estimate $\widehat{\mathcal{E}}$.

Choosing $\tau=5\phi_{n}(\delta)/Q^{*}(1-\varepsilon)$, we obtain that with probability at least $1-\delta$, the meshgrid $\widehat{\sigma}_{\tau}$ is finer than $\sigma_{\widetilde{\tau}_{1}}$ where $\widetilde{\tau}_{1}=\tau+(4\phi_{n}(\delta)+1/n)/Q^{*}(1-\varepsilon)$, and coarser than $\sigma_{\widetilde{\tau}_{0}}$ where $\widetilde{\tau}_{0}=\tau-4\phi_{n}(\delta)/Q^{*}(1-\varepsilon)$. Indeed, thanks to (34), $\mathcal{E}(I_{j,k})>\widetilde{\tau}_{1}$ implies that with probability $1-\delta$, $\widehat{\mathcal{E}}(I_{j,k})>\widetilde{\tau}_{1}-4\phi_{n}/Q^{*}(1-\varepsilon)>\tau$. It also implies that $j<j_{\text{max}}$. Indeed, if $j\geq j_{\text{max}}$, then $I_{j,k}=[\alpha_{1},\alpha_{2}]$ is such that $\alpha_{2}-\alpha_{1}<1/n$ which gives, using the mean value theorem, $\mathcal{E}(I_{j,k})=\lambda(\Omega_{\alpha_{2}})-\lambda(\Omega_{\alpha_{1}})<1/(Q^{*}(1-\varepsilon)n)$ which contradicts $\mathcal{E}(I_{j,k})>\widetilde{\tau}_{1}>1/(Q^{*}(1-\varepsilon)n$. Therefore with probability at least $1-\delta$, splitting $I_{j,k}$ for the true local error and a tolerance $\widetilde{\tau}_{1}$ implies splitting $I_{j,k}$ in Algorithm 2. Analogously, $\widehat{\mathcal{E}}(I_{j,k})>\tau$ implies, thanks to (34), that with probability at least $1-\delta$, $\mathcal{E}(I_{j,k})>\widetilde{\tau}_{0}$. Therefore with probability at least $1-\delta$, splitting $I_{j,k}$ in Algorithm 2 implies splitting $I_{j,k}$ for the true local error and a tolerance $\widetilde{\tau}_{0}$.

We now use the following bound

[TABLE]

The second term is bounded by $\widetilde{\tau}_{1}$ with probability at least $1-\delta$ because with probability at least $1-\delta$ the meshgrid $\widehat{\sigma}_{\tau}$ is finer than $\sigma_{\widetilde{\tau}_{1}}$ and therefore

[TABLE]

On the same event, for the first term we have,

[TABLE]

Thus we finally have

[TABLE]

For the second part of the theorem on the cardinality of the meshgrid $\widehat{\sigma}_{\tau}$ obtained with Algorithm 2 we use the following lemma (DeVore, 1998).

Lemma 8**.**

Let $\text{\emph{card}}(\sigma_{\tau})$ be the number of terminal nodes obtained when the algorithm is implemented with the true local error $\mathcal{E}(I)$. Suppose that assumptions (A1), (A2) and (C3) are fulfilled. There exists a universal constant $C>0$ such that, for all $\tau>0$:

[TABLE]

Applying this lemma to $\widetilde{\tau}_{0}=\phi_{n}(\delta)/Q^{*}(1-\varepsilon)$ we obtain that the number of terminal nodes $\text{card}(\sigma_{\widetilde{\tau}_{0}})$ of $\sigma_{\widetilde{\tau}_{0}}$ is such that

[TABLE]

As $\widehat{\sigma}_{\tau}$ is coarser than $\sigma_{\widetilde{\tau}_{0}}$ with probability at least $1-\delta$, the number of terminal nodes of the former is bounded by the number of terminal nodes of the latter with probability at least $1-\delta$.

A.3.2 Proof of Theorem 4

We use the following notation: $\widetilde{\alpha}_{k}=F(\widetilde{\Omega}_{\alpha_{k}})$. We first prove a lemma quantifying the loss coming from the monotonicity step of Algorithm 3.

Lemma 9**.**

Suppose that the assumptions of Theorem 4 are satisfied. Let $\delta\in(0,1)$. For all $n\geq 1$, with probability at least $1-\delta$, we have: $\forall k\in\{1,\dots,\widehat{K}\}$,

[TABLE]

and

[TABLE]

where $C_{1}>0$ and $C_{2}>0$ are constants depending only on $\gamma$, $\|f\|_{\infty}$ and the constant $C$ of assumption (C4).

Proof.

First, from proposition 4, we have with probability at least $1-\delta$, $\forall k\in\{0,\dots,\widehat{K}\}$,

[TABLE]

For $k=0$, we have $\alpha_{0}=0$ and thus $\widetilde{\Omega}_{\alpha_{0}}=\widehat{\Omega}_{\alpha_{0}}=\emptyset$ and the inequalities of the Lemma are trivially satisfied. For $k=1$, we have $\widetilde{\Omega}_{\alpha_{1}}=\widehat{\Omega}_{\alpha_{1}}$ and with (37) the inequalities of the Lemma are satisfied. For all $k\in\{0,\dots,\widehat{K}\}$,

[TABLE]

and $\widetilde{\alpha}_{k}-\alpha_{k}\geq-2\phi_{n}(\delta)$. The same argument gives

[TABLE]

We show how to obtain the upper bound for the case $k=2$. We have $\widetilde{\Omega}_{\alpha_{2}}=\widehat{\Omega}_{\alpha_{1}}\cup\widehat{\Omega}_{\alpha_{2}}$ and

[TABLE]

Now,

[TABLE]

where the last inclusion comes from the fact that $\Omega^{*}_{\alpha_{1}}\subset\Omega^{*}_{\alpha_{2}}$. Hence,

[TABLE]

and thus

[TABLE]

Using assumption (C4), for any $\alpha\in(0,1)$, if $\widehat{\Omega}_{\alpha}$ denotes the solution of the optimization problem (13), we can as in Polonik (1997) (see proof of Lemma 3.2) bound $F(\widehat{\Omega}_{\alpha}\Delta\Omega^{*}_{\alpha})$: with probability at least $1-\delta$,

[TABLE]

Minimizing the right-hand side with respect to $t$ gives

[TABLE]

where $C_{1}>0$ is a constant depending only on $\gamma$, $C$ and $\|f\|_{\infty}$. We thus finally have, with probability at least $1-\delta$,

[TABLE]

and therefore,

[TABLE]

For $k\geq 2$, we can obtain in a similar fashion,

[TABLE]

and derive the following result: with probability at least $1-\delta$, for all $k\geq 2$,

[TABLE]

For (36), first note that for all $\alpha\in[0,1-\varepsilon]$, $\lambda(\Omega^{*}_{\alpha}\Delta\widehat{\Omega}_{\alpha})\leq C_{2}\phi_{n}(\delta)^{\frac{\gamma}{\gamma+1}}$ where $C_{2}>0$ depends only on $\gamma$ and $C$. Therefore the same procedure leads to: with probability $1-\delta$, for all $k\geq 0$,

[TABLE]

∎

We can now prove Theorem 4. We first write

[TABLE]

Taking $\tau=4\phi_{n}(\delta)/Q^{*}(1-\varepsilon)+c_{n}$, $c_{n}>0$ and using the same argument as in the proof of Theorem 3, the second term of the right-hand side is bounded by $\widetilde{\tau}_{1}=\tau+(4\phi_{n}(\delta)+1/n)/Q^{*}(1-\varepsilon)$. We now deal with the first term. Let $\alpha\in[0,1-\varepsilon]$,

[TABLE]

But one can observe that

[TABLE]

to obtain

[TABLE]

Now $\alpha\mapsto E_{\widehat{\sigma}_{\tau}}({\rm MV}^{*})(\alpha)$ is increasing, therefore on $[\widetilde{\alpha}_{k},\widetilde{\alpha}_{k+1})$, $E_{\widehat{\sigma}_{\tau}}({\rm MV}^{*})(\alpha)\geq E_{\widehat{\sigma}_{\tau}}({\rm MV}^{*})(\widetilde{\alpha}_{k})$ and

[TABLE]

We thus have

[TABLE]

The first term is equal to $\max_{0\leq k\leq\widehat{K}-1}|\lambda(\widetilde{\Omega}_{\alpha_{k+1}})-{\rm MV}^{*}(\alpha_{k+1})|$ and thanks to Lemma 9 we have with probability at least $1-\delta$

[TABLE]

For the second term we write

[TABLE]

where $j_{0}\in\{0,\dots,\widehat{K}-1 \}$ is such that $\widetilde{\alpha}_{k}\in[\alpha_{j_{0}},\alpha_{j_{0}+1})$. We bound $|{\rm MV}^{*}(\alpha_{k+1})-{\rm MV}^{*}(\alpha_{j_{0}+1})|$ by

[TABLE]

We bound the first term as follows. Let $I=[\alpha_{k},\alpha_{k+1}]$. If the depth $j_{\text{max}}$ has been reached, $\alpha_{k+1}-\alpha_{k}<1/n$. Therefore the mean value theorem gives:

[TABLE]

Otherwise, this means that $\widehat{\mathcal{E}}(I)\leq\tau$ and we decompose $|{\rm MV}^{*}(\alpha_{k+1})-{\rm MV}^{*}(\alpha_{k})|=\mathcal{E}(I)$ into

[TABLE]

Using Lemma 9, the right-hand side is bounded by $4\phi_{n}(\delta)/Q^{*}(1-\varepsilon)+\tau$. Eventually,

[TABLE]

Using the mean value theorem and Lemma 9, the second term can be bounded by

[TABLE]

As $\widetilde{\alpha}_{k}\in[\alpha_{j_{0}},\alpha_{j_{0}+1})$ and as ${\rm MV}^{*}$ is increasing, the third term is bounded by $|\lambda(\Omega^{*}_{\alpha_{j_{0}+1}})-\lambda(\Omega^{*}_{\alpha_{j_{0}}})|$ which is also bounded as the first term by $\max(4\phi_{n}(\delta)/Q^{*}(1-\varepsilon)+\tau,1/(nQ^{*}(1-\varepsilon)))$. We thus obtain

[TABLE]

and combining this inequality with (38), we have

[TABLE]

Eventually, with probability at least $1-\delta$,

[TABLE]

Using the same argument as in the proof of Theorem 3, on the same event, which holds with probability at least $1-\delta$, $\widehat{\sigma}_{\tau}$ is coarser than $\sigma_{\widetilde{\tau_{0}}}$ where $\widetilde{\tau_{0}}=\tau-4\phi_{n}(\delta)/Q^{*}(1-\varepsilon)=c_{n}$ and therefore $\widehat{K}$ is of the order of $c_{n}$. Using assumption (C2), for $n$ large enough, $\phi_{n}(\delta)\leq\phi_{n}(\delta)^{\gamma/(\gamma+1)}$. Therefore the right-hand side of (39) is of the order of:

[TABLE]

Choosing $c_{n}=(1/n)^{\gamma/(4(1+\gamma))}$ to balance the two terms of this expression, we finally obtain that with probability at least $1-\delta$,

[TABLE]

Appendix B Technical results

This section contains the proofs of technical results that have been postponed for the sake of clarity.

B.1 Proof of Lemma 1

Proof.

We recall here that $\mathcal{Z}_{f}=\{x,F_{f}(f(x))>0\}$ and $\mathop{\rm supp}(f)=\{x,f(x)>0\}$. If $f(x)=0$ then $F_{f}(f(x))=0$ thus $\overline{\mathop{\rm supp}(f)}\subset\overline{\mathcal{Z}_{f}}$ and $\overline{\mathcal{Z}_{f}}\Delta\overline{\mathop{\rm supp}(f)}=\overline{\mathcal{Z}_{f}}\setminus\overline{\mathop{\rm supp}(f)}=\overline{\mathcal{Z}_{f}}\cap\mathop{\rm supp}(f)$. Therefore we have to prove $\lambda(\overline{\mathcal{Z}_{f}}\cap\mathop{\rm supp}(f))=0$. Let’s assume $\lambda(\overline{\mathcal{Z}_{f}}\cap\mathop{\rm supp}(f))>0$ and let $f^{*}=\sup_{u\in\overline{\mathcal{Z}_{f}}\cap\mathop{\rm supp}(f)}f(u)$. The intersection $\overline{\mathcal{Z}_{f}}\cap\mathop{\rm supp}(f)$ is not empty. Otherwise we would have $\lambda(\overline{\mathcal{Z}_{f}}\cap\mathop{\rm supp}(f))=0$. Therefore there exists $u\in\overline{\mathcal{Z}_{f}}\cap\mathop{\rm supp}(f)$ and we have $f^{*}\geq f(u)>0$ as $u\in\mathop{\rm supp}(f)$. We also have $F_{f}(f^{*})=0$. Indeed, $f^{*}$ is in the closure of $\{f(u),u\in\overline{\mathcal{Z}_{f}}\cap\mathop{\rm supp}(f)\}$ and therefore there exists a sequence $(f_{m})_{m\geq 0}$ of elements of $\{f(u),u\in\overline{\mathcal{Z}_{f}}\cap\mathop{\rm supp}(f)\}$ which converges towards $f^{*}$ as $m$ tends to infinity. Thus $F_{f}(f_{m})=0$ for all $m\geq 0$ and as $F_{f}$ is continuous, $F_{f}(f^{*})=0$, i.e.

[TABLE]

This implies that $f$ is equal to 0 $\lambda$-almost everywhere on $\{u,0\leq f(u)\leq f^{*}\}$. However, if $u\in\overline{\mathcal{Z}_{f}}\cap\mathop{\rm supp}(f)$ then, by definition of $f^{*}$, $f(u)\leq f^{*}$. Therefore $\overline{\mathcal{Z}_{f}}\cap\mathop{\rm supp}(f)\subset\{u,0\leq f(u)\leq f^{*}\}$ and as $\lambda(\overline{\mathcal{Z}_{f}}\cap\mathop{\rm supp}(f))>0$ and $f>0$ on $\mathop{\rm supp}(f)$, $\{u,0\leq f(u)\leq f^{*}\}$ contains a subset of non null $\lambda$-measure on which $f>0$. This is a contradiction with the fact that $f$ is equal to 0 $\lambda$-almost everywhere on $\{u,0\leq f(u)\leq f^{*}\}$ and thus $\lambda(\overline{\mathcal{Z}_{f}}\Delta\overline{\mathop{\rm supp}(f)})=0$, which is equivalent to $\lambda(\mathcal{Z}_{f}\Delta\mathop{\rm supp}(f))=0$. ∎

B.2 Proof of Lemma 3

Proof of Lemma 3.

The proof is based on Remark 3.2.4 in (Csörgő, 1983). Let $q_{n}$ denote the quantile process defined for all $\alpha\in[\varepsilon,1-\varepsilon]$ by

[TABLE]

Using the mean value theorem we can write for all $\alpha\in[\varepsilon,1-\varepsilon]$,

[TABLE]

where $\xi$ is between $\widehat{U}^{{\dagger}}(\alpha)$ and $\alpha$ and where $\widehat{U}^{{\dagger}}(\alpha)$ is defined as in the proof of Lemma 2. Let $u_{n}$ be the uniform quantile process defined in Lemma 2. For all $\alpha\in[\varepsilon,1-\varepsilon]$, $q_{n}(\alpha)=u_{n}(\alpha)f_{s}(F_{s}^{{\dagger}}(\alpha))/f_{s}(F_{s}^{{\dagger}}(\xi))$.

Now we know from Theorem 1 in (Csörgő and Révész, 1978) that for all $z_{1}$ and for all $n$,

[TABLE]

where $C_{1},C_{2}$ and $C_{3}$ are positive absolute constants and where $\{B_{n}^{1}(\alpha),\alpha\in[0,1]\}_{n\geq 1}$ is the same sequence of Brownian Bridges as the one of the proof of assertion $(ii)$ of Theorem 1. For all $\alpha\in[\varepsilon,1-\varepsilon]$, we can write

[TABLE]

Using a Taylor expansion of $\lambda_{s}$ as in the proof of part $(ii)$ of Theorem 1 we have for all $\alpha\in[\varepsilon,1-\varepsilon]$,

[TABLE]

where $\xi_{1}$ is between $\widehat{\alpha}_{s}^{-1}(\alpha)$ and $\alpha_{s}^{-1}(\alpha)$. With the change of variable $\alpha^{\prime}=1-\alpha$, we obtain for all $\alpha^{\prime}\in[\varepsilon,1-\varepsilon]$ that $\xi_{1}$ is between $\widehat{F}_{s}^{{\dagger}}(\alpha^{\prime})$ and $F_{s}^{{\dagger}}(\alpha^{\prime})$ and that

[TABLE]

and therefore

[TABLE]

To bound the right-hand side of the previous inequality we need the following results. From the DKW inequality (see proof of Lemma 2) we have for all $z_{2}>0$ and for all $n$,

[TABLE]

From Theorem 1.5.1 in (Csörgő, 1983) we have for all $z_{3}>0$ and for all $n$,

[TABLE]

where for all $z>0$, $h(z)=z+\log(1/z)-1$ and where $x\mapsto\lfloor x\rfloor$ denotes the floor part function. Finally from the proof of Theorem 1.4.3 in (Csörgő, 1983) we have for all $z_{4}\geq 1$ and for all $n$,

[TABLE]

Let $z_{1},z_{2}$ and $z_{3}$ be positive and let $z_{4}\geq 1$. We consider the four following events:

[TABLE]

Let $\omega\in\overline{\mathcal{Z}_{1}}\cap\overline{\mathcal{Z}_{2}}\cap\overline{\mathcal{Z}_{3}}\cap\overline{\mathcal{Z}_{4}}$ where $\overline{\mathcal{Z}}$ denotes the complement of a set $\mathcal{Z}$. We first deal with the first term of the right-hand side of (42). From (41) we have for all $\alpha^{\prime}\in[\varepsilon,1-\varepsilon]$,

[TABLE]

where the second inequality holds because $\omega\in\overline{\mathcal{Z}_{1}}\cap\overline{\mathcal{Z}_{2}}\cap\overline{\mathcal{Z}_{3}}$. Now for all $\alpha^{\prime}\in[\varepsilon,1-\varepsilon]$, $F_{s}^{{\dagger}}(\alpha^{\prime})\in[0,\|s\|_{\infty}]$. As $\lambda_{s}^{\prime}$ is continuous $|\lambda_{s}^{\prime}(F_{s}^{{\dagger}}(\alpha^{\prime}))|\leq\sup_{[0,\|s\|_{\infty}]}|\lambda_{s}^{\prime}|<\infty$. Furthermore, for all $\alpha^{\prime}\in[\varepsilon,1-\varepsilon]$, $f_{s}(F_{s}^{{\dagger}}(\alpha^{\prime}))\geq\inf_{\alpha\in[\varepsilon,1-\varepsilon]}f_{s}(F_{s}^{{\dagger}}(\alpha^{\prime}))$ where the infimum is strictly positive. Therefore for all $\alpha^{\prime}\in[\varepsilon,1-\varepsilon]$,

[TABLE]

We now deal with the second term of the right-hand side of (42). For all $\alpha\in[\varepsilon,1-\varepsilon]$, $\alpha_{s}^{-1}(\alpha)$ and $\widehat{\alpha}_{s}^{-1}(\alpha)$ are in $[0,\|s\|_{\infty}]$. Therefore $\xi_{1}\in[0,\|s\|_{\infty}]$ and as $\lambda_{s}^{\prime\prime}$ is continuous, $|\lambda_{s}^{\prime\prime}(\xi_{1})|\leq\sup_{[0,\|s\|_{\infty}]}|\lambda_{s}^{\prime\prime}|$. Now, as in the proof of Lemma 2, using the mean value theorem we can write for $\alpha^{\prime}\in[\varepsilon,1]$,

[TABLE]

where $\xi$ is the one of (41). As $\omega\in\overline{\mathcal{Z}_{2}}\cap\overline{\mathcal{Z}_{4}}$ and bounding $1/f_{s}(F_{s}^{{\dagger}}(\alpha^{\prime}))$ by $1/\inf_{[\varepsilon,1-\varepsilon]}f_{s}\circ F_{s}^{{\dagger}}$ as previously we obtain

[TABLE]

We eventually have

[TABLE]

Choosing, $z_{1}=c_{1}\log n$ with $c_{1}>0$ arbitrary, $z_{2}=\sqrt{\log n}$, $z_{3}=c_{3}\sqrt{\log n/n}$ with $c_{3}>0$ arbitrary, and $z_{4}=2$ we obtain that,

[TABLE]

where the constant $c_{4}$ is given by

[TABLE]

Let $\mathcal{Z}_{5}$ be the event defined by

[TABLE]

We have shown that $\omega\in\overline{\mathcal{Z}_{5}}$ therefore

[TABLE]

We can choose $c_{1}$ such that $n^{-C_{3}c_{1}}=O(v_{n})$. Studying the function $h$ we can also show that $h(2^{1/2(\lfloor c\rfloor+1)})$ and $h(2^{-1/2(\lfloor c\rfloor+1)})$ are strictly positive and if $a=1/2(\lfloor c\rfloor+1)$, as $n\rightarrow\infty$,

[TABLE]

Similarly we can show that

[TABLE]

and we can choose $c_{3}$ such that the right-hand sides of the two previous equations are of order $O(v_{n})$. This finally gives the result of the lemma. ∎

B.3 Proof of Lemma 4

Proof of Lemma 4.

To prove this lemma we carefully follow the proof of Lemma 3. From Lemma 11, Lemma 12 and Lemma 14 we know that there exist constants $C_{1}$ and $C_{2}$ both strictly positive such that for $\mathbb{P}$-almost all $\mathcal{D}_{n}$, for $n$ large enough, $\sup_{t\in\mathbb{R}}|\widetilde{f}_{s}(t)-f_{s}(t)|\leq C_{1}w_{n}$, $\sup_{\alpha\in[\varepsilon,1-\varepsilon]}|\widetilde{F_{s}}^{{\dagger}}(\alpha)-F_{s}^{{\dagger}}(\alpha)|\leq C_{2}w_{n}$ and $\sup_{t\in\mathbb{R}}|\widetilde{f_{s}}^{\prime}(t)-f_{s}^{\prime}(t)|$ converges to [math] as $n$ tends to $\infty$. Let $\mathcal{D}_{n}$ be such that these assertions are fulfilled. We denote by $q_{n}^{*}$ the quantile process given for all $\alpha^{\prime}\in(0,1)$ by

[TABLE]

where $(F_{s}^{Boot})^{{\dagger}}$ is defined on $(0,1)$ as the generalized inverse of the empirical cumulative distribution function $F_{s}^{Boot}$ which is based on a bootstrap sample $S_{1}^{*},\dots,S_{n}^{*}$ which are i.i.d. random variables with distribution $\widetilde{F}_{s}$.

We have for all $\alpha\in[\varepsilon,1-\varepsilon]$,

[TABLE]

where $\widetilde{U}^{{\dagger}}(\alpha)=\widetilde{F}_{s}((F_{s}^{Boot})^{{\dagger}}(\alpha))$ for all $\alpha\in(0,1)$. Let $u^{*}_{n}$ be the associated uniform quantile process defined for all $\alpha\in(0,1)$ by $u_{n}^{*}(\alpha)=\sqrt{n}\left(\widetilde{U}^{{\dagger}}(\alpha)-\alpha\right)$.

Let $\varepsilon^{\prime}\in(0,1)$. We first prove the differentiability of $\widetilde{F}_{s}^{{\dagger}}$ on $[\varepsilon^{\prime},1-\varepsilon^{\prime}]$ for $n$ large enough. Given assumption (B5) the kernel $K$ is differentiable on $\mathbb{R}$. Therefore $\widetilde{f}_{s}$ is differentiable on $\mathbb{R}$. For $n$ large enough and for all $\alpha\in[\varepsilon^{\prime},1-\varepsilon^{\prime}]$, $\widetilde{f_{s}}(\widetilde{F}_{s}^{{\dagger}}(\alpha))=\widetilde{f_{s}}(\widetilde{\alpha}_{s}^{-1}(1-\alpha))>0$. Indeed, we can write

[TABLE]

As $\sup_{\alpha\in[\varepsilon^{\prime},1-\varepsilon^{\prime}]}|\widetilde{F_{s}}^{{\dagger}}(\alpha)-F_{s}^{{\dagger}}(\alpha)|$ converges to [math], $\widetilde{\alpha}_{s}^{-1}(\varepsilon^{\prime})$ and $\widetilde{\alpha}_{s}^{-1}(1-\varepsilon^{\prime})$ converge respectively towards $\alpha_{s}^{-1}(\varepsilon^{\prime})$ and $\alpha_{s}^{-1}(1-\varepsilon^{\prime})$. Therefore there exists a constant $c_{1}>0$, depending only on $\alpha_{s}^{-1}(1-\varepsilon^{\prime})$ and $\alpha_{s}^{-1}(\varepsilon^{\prime})$, such that for $n$ large enough, $\widetilde{\alpha_{s}}^{-1}(1-\varepsilon^{\prime})\geq\alpha_{s}^{-1}(1-\varepsilon^{\prime})-c_{1}>a$ and $\widetilde{\alpha_{s}}^{-1}(\varepsilon^{\prime})<\alpha_{s}^{-1}(\varepsilon^{\prime})+c_{1}<b$, i.e., $[\widetilde{\alpha_{s}}^{-1}(1-\varepsilon^{\prime}),\widetilde{\alpha_{s}}^{-1}(\varepsilon^{\prime})]\subset[\alpha_{s}^{-1}(1-\varepsilon^{\prime})-c_{1},\alpha_{s}^{-1}(\varepsilon^{\prime})+c_{1}]$. Therefore for $n$ large enough,

[TABLE]

and as $\widetilde{f}_{s}$ converges uniformly towards $f_{s}$, there exists a constant $c_{2}>0$ such that for $n$ large enough,

[TABLE]

Note that $c_{2}$ can be chosen such that the strict positivity holds because the infimum of $f_{s}$ on $[\alpha_{s}^{-1}(1-\varepsilon^{\prime})-c_{1},\alpha_{s}^{-1}(\varepsilon^{\prime})+c_{1}]$ is attained and thus strictly positive (assumption (A4)). Eventually, combining the three previous inequalities, for $n$ large enough we have: for all $\alpha\in[\varepsilon^{\prime},1-\varepsilon^{\prime}]$, $\widetilde{f}_{s}(\widetilde{\alpha}_{s}^{-1}(\alpha))>0$.

Therefore, $\widetilde{F}_{s}$ is differentiable on $[\widetilde{\alpha}_{s}^{-1}(1-\varepsilon^{\prime}),\widetilde{\alpha}_{s}^{-1}(\varepsilon^{\prime})]$ with strictly positive derivative. This gives that $\widetilde{F}_{s}^{{\dagger}}:[\varepsilon^{\prime},1-\varepsilon^{\prime}]\rightarrow[\widetilde{\alpha}_{s}^{-1}(1-\varepsilon^{\prime}),\widetilde{\alpha}_{s}^{-1}(\varepsilon^{\prime})]$ is equal to the ordinary inverse of $\widetilde{F}_{s}$ on $[\varepsilon^{\prime},1-\varepsilon^{\prime}]$ and $\widetilde{F}_{s}^{{\dagger}}$ is differentiable on $[\varepsilon^{\prime},1-\varepsilon^{\prime}]$.

We now need to invoke the same argument as in the beginning of the proof of Lemma 16. This argument says that there exists a constant $c_{3}$ such that for $n$ large enough, if $\mathcal{Z}_{0}$ denotes the event $\mathcal{Z}_{0}=\{[\widetilde{U}^{{\dagger}}(\varepsilon),\widetilde{U}^{{\dagger}}(1-\varepsilon)]\not\subset[\varepsilon-c_{3},1-\varepsilon+c_{3}]\}$ then $\mathbb{P}^{*}(\mathcal{Z}_{0})\leq 2/n^{2}$ (refer to the proof of Lemma 16 for details). Let $\omega\in\overline{\mathcal{Z}}_{0}$. Assuming $n$ large enough, for all $\alpha\in[\varepsilon,1-\varepsilon]$, $\widetilde{U}^{{\dagger}}(\alpha)\in[\varepsilon-c_{3},1-\varepsilon+c_{3}]$. Furthermore for $n$ large enough, we proved that $\widetilde{F}_{s}^{{\dagger}}$ is differentiable on $[\varepsilon^{\prime},1-\varepsilon^{\prime}]$ for all $\varepsilon^{\prime}\in(0,1)$. Therefore we can apply the mean value theorem to write for all $\alpha\in[\varepsilon,1-\varepsilon]$,

[TABLE]

where $\xi$ is between $\widetilde{U}^{{\dagger}}(\alpha^{\prime})$ and $\alpha^{\prime}$.

Applying Theorem 1 in (Csörgő and Révész, 1978) conditionally on $\mathcal{D}_{n}$, there exists a sequence of Brownian Bridges $\{B^{*1}_{n}(\alpha),\alpha\in[0,1]\}_{n\geq 1}$ such that for all $z_{1}$ and for all $n$,

[TABLE]

where $C_{3},C_{4}$ and $C_{5}$ are positive absolute constants. Let $B_{n}^{*}(\alpha)=B_{n}^{*1}(1-\alpha)$ for all $\alpha\in[0,1]$. Reasoning similarly as in the proof of Lemma 3, we have for all $\alpha^{\prime}\in[\varepsilon,1-\varepsilon]$,

[TABLE]

where $\xi_{1}$ is between $(F_{s}^{Boot})^{{\dagger}}(\alpha^{\prime})$ and $\widetilde{F}_{s}^{{\dagger}}(\alpha^{\prime})$.

Still following the proof of Lemma 3 we need the following results. Conditionally on $\mathcal{D}_{n}$, the DKW inequality gives for all $z_{2}>0$ and for all $n$,

[TABLE]

Using the result of Lemma 16 there exists a constant $c^{*}>0$ such that we have, conditionally on $\mathcal{D}_{n}$, for all $z_{4}\geq 1$ and for $n$ large enough,

[TABLE]

where we recall that for all $z>0$, $h(z)=z+\log(1/z)-1$. Following the proof of Lemma 16 and with calculations similar as the ones used in the proof of Theorem 1.4.3 in (Csörgő, 1983) it can also be shown that conditionally on $\mathcal{D}_{n}$ we have for all $z_{3}>0$ and for $n$ large enough,

[TABLE]

Now as for all $\alpha^{\prime}\in[\varepsilon,1-\varepsilon]$, $\widetilde{F}_{s}^{{\dagger}}(\alpha^{\prime})$ is in the support of $\widetilde{f}_{s}$ which has shown in the proof of Lemma 13, is included in an interval of the form $[-c_{4},c_{4}+\|s\|_{\infty}]$ where $c_{4}$ is a strictly positive constant that depends only on the kernel $K$. As $\lambda_{s}^{\prime}$ is continuous, $|\lambda_{s}^{\prime}(\widetilde{F}_{s}^{{\dagger}}(\alpha^{\prime}))|\leq\sup_{[-c_{4},c_{4}+\|s\|_{\infty}]}|\lambda_{s}^{\prime}|<\infty$. Furthermore, as proved above, there exist constants $c_{5}$ and $c_{6}$ both strictly positive such that $[F_{s}^{{\dagger}}(\varepsilon)-c_{5},F_{s}^{{\dagger}}(1-\varepsilon)+c_{5}]\subset(a,b)$ and such that for $n$ large enough,

[TABLE]

For all $\alpha\in[\varepsilon,1-\varepsilon]$, $(F_{s}^{Boot})^{{\dagger}}(\alpha)$ and $\widetilde{F}_{s}^{{\dagger}}(\alpha)$ are in the support of $\widetilde{f}_{s}$ which is included in $[-c_{4},c_{4}+\|s\|_{\infty}]$. Therefore $\xi_{1}\in[-c_{4},c_{4}+\|s\|_{\infty}]$ and as $\lambda_{s}^{\prime\prime}$ is continuous, $|\lambda_{s}^{\prime\prime}(\xi_{1})|\leq\sup_{[-c_{4},c_{4}+\|s\|_{\infty}]}|\lambda_{s}^{\prime\prime}|<+\infty$. Thus following the steps of the proof of Lemma 3 we have for $n$ large enough,

[TABLE]

where

[TABLE]

Proceeding analogously as in Lemma 3, we can choose $c_{7}$ and $c_{8}$ such that the right-hand side of the previous inequality is of order $O(v_{n})$. This finally gives the result of the lemma. ∎

B.4 Proof of Lemma 5

Proof of Lemma 5.

Notice first that thanks to Lemma 10 $y_{s}(\alpha)\neq 0$ for all $\alpha\in(0,1)$. Let $\alpha\in[\varepsilon,1-\varepsilon]$,

[TABLE]

As the function $|1/y_{s}(\cdot)|$ is continuous and therefore bounded over $[\varepsilon,1-\varepsilon]$, we only need to control $\sup_{\alpha\in[\varepsilon,1-\varepsilon]}|\widetilde{y}_{s}(\alpha)-y_{s}(\alpha)|$ which can be bounded by the following decomposition

[TABLE]

As $\lambda_{s}$ is of class $\mathcal{C}^{2}$ and $f_{s}$ is of class $\mathcal{C}^{1}$ and strictly positive on $(a,b)$, $\lambda_{s}^{\prime}/f_{s}$ is of class $\mathcal{C}^{1}$ on $(a,b)$. Thus using the mean value theorem, we can write, for all $\alpha\in[\varepsilon,1-\varepsilon]$,

[TABLE]

where $\xi$ is between $\alpha_{s}^{-1}(\alpha)$ and $\widetilde{\alpha}_{s}^{-1}(\alpha)$. From Lemma 14 we know that there exists a constant $c_{1}>0$ such that almost surely, for $n$ large enough, $\widetilde{\alpha}_{s}^{-1}(\alpha)\in[\alpha_{s}^{-1}(1-\varepsilon)-c_{1},\alpha_{s}^{-1}(\varepsilon)+c_{1}]\subset(a,b)$. Therefore, as $(\lambda_{s}^{\prime}/f_{s})^{\prime}$ is continuous, almost surely and for $n$ large enough,

[TABLE]

Furthermore,

[TABLE]

Therefore, using Lemma 14, there exists a constant $C_{1}>0$ such that we almost surely have, for $n$ large enough,

[TABLE]

For the second term of (51) we write

[TABLE]

and as shown in the proof of Lemma 13, $\widetilde{F}_{s}$ has a finite support included in an interval of the form $[-c_{2},c_{2}+\|s\|_{\infty}]$. Therefore $\widetilde{\alpha}_{s}^{-1}(\alpha)\in[-c_{2},c_{2}+\|s\|_{\infty}]$ and as $\lambda_{s}^{\prime}$ is continuous, it is bounded on this interval. Therefore there exists $C_{2}>0$ such that for all $\alpha\in[\varepsilon,1-\varepsilon]$, $|\lambda^{\prime}_{s}(\widetilde{\alpha}_{s}^{-1}(\alpha))|\leq C_{2}$ and

[TABLE]

We can proceed as in the proof of Lemma 4 to show that there exists constants $c_{3}$ and $c_{4}$ both strictly positive such that $[\alpha_{s}^{-1}(1-\varepsilon)-c_{3},\alpha_{s}^{-1}(\varepsilon)+c_{3}]\subset(a,b)$ and such that almost surely, for $n$ large enough, for all $\alpha\in[\varepsilon,1-\varepsilon]$,

[TABLE]

We also almost surely have, for $n$ large enough and for all $\alpha\in[\varepsilon,1-\varepsilon]$,

[TABLE]

Let $A_{\varepsilon}=[\alpha_{s}^{-1}(1-\varepsilon)-c_{3},\alpha_{s}^{-1}(\varepsilon)+c_{3}]$. We almost surely have, for $n$ large enough,

[TABLE]

Thanks to Lemma 11 and combining the last inequality with (52), we obtain the result of the lemma. ∎

B.5 Other technical results

Lemma 10**.**

For all $\alpha\in(0,1)$, $y_{s}(\alpha)\neq 0$.

Proof of Lemma 10.

First notice that as $\lambda_{s}$ is of class $\mathcal{C}^{2}$ and $F_{s}^{\prime}=f_{s}>0$ on $(a,b)$, ${\rm MV}_{s}=\lambda_{s}\circ\alpha_{s}^{-1}$ is differentiable on $(0,1)$ and for all $\alpha\in(0,1)$,

[TABLE]

Now, following the steps of the proof of the formula of the derivative of ${\rm MV}^{*}$ we also have, for all $\alpha\in(0,1)$ and $h>0$,

[TABLE]

where we used the fact that the distribution of $F_{s}(s(X))$ is the uniform distribution on $(0,1)$. Similarly,

[TABLE]

Therefore, for all $\alpha\in(0,1)$, ${\rm MV}_{s}^{\prime}(\alpha)\geq 1/\|f\|_{\infty}>0$. Using (54), this implies that $y_{s}(\alpha)\neq 0$ for all $\alpha\in(0,1)$. ∎

Lemma 11**.**

Under assumptions (A3) and (B1)-(B4) there exists a constant $C>0$, depending only on $f_{s}$ and the kernel $K$ such that we almost surely have, for $n$ large enough,

[TABLE]

Note that this result does not require $K$ to have a finite support.

Proof of Lemma 11.

We first bound $\sup_{t\in\mathbb{R}}|\widetilde{f}_{s}(t)-f_{s}(t)|$ by the sum of a stochastic term and a deterministic term:

[TABLE]

Under the stipulated conditions, the first term is controlled thanks to Theorem 2.3 in (Giné and Guillou, 2002): we almost surely have, for $n$ large enough,

[TABLE]

where $M$ is a constant that depends on the VC characteristics of $K$.

It can then be shown with classical calculations that the second term is bounded by

[TABLE]

Combining this last inequality with the one on the first term we obtain the result of the lemma. ∎

Lemma 12**.**

Under assumptions (A3), (B1)-(B3) and (B5) there exists a constant $C>0$, depending only on $f_{s}$ and the kernel $K$ such that we almost surely have, for $n$ large enough,

[TABLE]

Therefore, as $h_{n}\rightarrow 0$, if assumption (B6) is also fulfilled then we almost surely have,

[TABLE]

Note that this result does not require $K$ to have a finite support.

Proof of Lemma 12.

We proceed as in the proof of Lemma 11. We first bound $\sup_{t\in\mathbb{R}}|\widetilde{f_{s}}^{\prime}(t)-f_{s}^{\prime}(t)|$ by the sum of a stochastic term and a deterministic term:

[TABLE]

We first deal with the first term. Let $t\in\mathbb{R}$,

[TABLE]

Therefore $K^{\prime}$ satisfying the conditions required for the result of (Giné and Guillou, 2002) we almost surely have, for $n$ large enough,

[TABLE]

where $M^{\prime}$ is a constant that depends on the VC characteristics of $K^{\prime}$.

We now deal with the second term as follows. Let $t\in\mathbb{R}$,

[TABLE]

With an integration by parts we obtain,

[TABLE]

and we can deal with this term as we dealt with $\mathbb{E}[\widetilde{f_{s}}(t)]-f_{s}(t)$ in the proof of Lemma 11, to obtain

[TABLE]

Combining this last inequality with the one on the first term we obtain the result of the lemma.

∎

Lemma 13**.**

Under the conditions of Lemma 11 there exists a constant $C>0$ such that we almost surely have for $n$ large enough,

[TABLE]

Proof of Lemma 13.

Let $t\in\mathbb{R}$,

[TABLE]

As $f_{s}$ has a support included in $[0,\|s\|_{\infty}]$, as $K$ has a finite support and as the sequence $(h_{n})_{n\geq 1}$ decreases towards [math], $\widetilde{f}_{s}$ also has a finite support included in an interval of the form $[-c_{1},\|s\|_{\infty}+c_{1}]$ where $c_{1}>0$. Thus,

[TABLE]

Thanks to Lemma 11, we finally obtain the result. ∎

Lemma 14**.**

Let $\varepsilon\in(0,1)$. Under the conditions of Lemma 11, there exists a constant $C>0$ such that we almost surely have for $n$ large enough,

[TABLE]

This lemma only requires $\widetilde{F}_{s}$ to be continuous.

Proof of Lemma 14.

First we recall that $\widetilde{F}_{s}$ being continuous, for all $y\in(0,1)$, $F_{s}(F_{s}^{{\dagger}}(y))=y$. Let $y\in[\varepsilon,1-\varepsilon]$,

[TABLE]

with $\xi$ between $F_{s}^{{\dagger}}(y)$ and $\widetilde{F}_{s}^{{\dagger}}(y)$. As $F_{s}^{{\dagger}}$ and $\widetilde{F}_{s}^{{\dagger}}$ are increasing, $F_{s}^{{\dagger}}(\varepsilon)\leq F_{s}^{{\dagger}}(y)\leq F_{s}^{{\dagger}}(1-\varepsilon)$ and $\widetilde{F}_{s}^{{\dagger}}(\varepsilon)\leq\widetilde{F}_{s}^{{\dagger}}(y)\leq\widetilde{F}_{s}^{{\dagger}}(1-\varepsilon)$. Now

[TABLE]

Therefore from Lemma 13 as $w_{n}\rightarrow 0$ when $n$ tends to $+\infty$, $F_{s}(\widetilde{F}_{s}^{{\dagger}}(\varepsilon))$ converges almost surely to $\varepsilon$. And as $F_{s}^{{\dagger}}$ is continuous, we almost surely have, $F_{s}^{{\dagger}}(F_{s}(\widetilde{F}_{s}^{{\dagger}}(\varepsilon)))\underset{n\rightarrow\infty}{\longrightarrow}F_{s}^{{\dagger}}(\varepsilon)$, i.e., $\widetilde{F}_{s}^{{\dagger}}(\varepsilon)\underset{n\rightarrow\infty}{\longrightarrow}F_{s}^{{\dagger}}(\varepsilon)$. Similarly, we almost surely have $\widetilde{F}_{s}^{{\dagger}}(1-\varepsilon)\underset{n\rightarrow\infty}{\longrightarrow}F_{s}^{{\dagger}}(1-\varepsilon)$. Therefore there exists a constant $c_{1}>0$ such that we almost surely have, for $n$ large enough, $\widetilde{F}_{s}^{{\dagger}}(y)\in[F_{s}^{{\dagger}}(\varepsilon)-c_{1},F_{s}^{{\dagger}}(1-\varepsilon)+c_{1}]\subset(a,b)$. This implies that $\xi$ is between $F_{s}^{{\dagger}}(\varepsilon)-c_{1}>a$ and $F_{s}^{{\dagger}}(1-\varepsilon)+c_{1}<b$ and thus,

[TABLE]

We thus almost surely have, for $n$ large enough,

[TABLE]

Now as $f_{s}$ is strictly positive on $(a,b)$ and continuous, the infimum of the right-hand side is strictly positive. We can thus write almost surely and for $n$ large enough,

[TABLE]

where we used Lemma 13 for the last inequality. This concludes the proof. ∎

Lemma 15**.**

Let $a^{\prime},b^{\prime}\in\mathbb{R}$ with $a<a^{\prime}<b^{\prime}<b$. Under assumptions (A3)-(A4) and (B1)-(B6),

[TABLE]

Proof of Lemma 15.

We have for all $t\in[a^{\prime},b^{\prime}]$,

[TABLE]

where the third inequality holds because for all $t$, $\widetilde{F}_{s}(t)\in[0,1]$. The result can then be derived thanks to Lemma 11, Lemma 12 and Lemma 13. ∎

Lemma 16**.**

Let $\mathcal{D}_{n}$ be such that $\sup_{t\in\mathbb{R}}|\widetilde{f}_{s}(t)-f_{s}(t)|\leq C_{1}w_{n}$, $\sup_{\alpha\in[\varepsilon,1-\varepsilon]}|\widetilde{F_{s}}^{{\dagger}}(\alpha)-F_{s}^{{\dagger}}(\alpha)|\leq C_{2}w_{n}$ and $\sup_{t\in\mathbb{R}}|\widetilde{f_{s}}^{\prime}(t)-f_{s}^{\prime}(t)|$ converges to [math] as $n$ tends to $\infty$. There exists a constant $c^{*}>0$ such that at $\mathcal{D}_{n}$ fixed, for all $z_{4}\geq 1$ and for $n$ large enough,

[TABLE]

where $\xi$ is between $\widetilde{U}^{{\dagger}}(\alpha)$ and $\alpha$ with $\alpha\in[\varepsilon,1-\varepsilon]$.

Proof of Lemma 16.

From the DKW inequality (see proof of Lemma 2), we have for all $n$,

[TABLE]

Therefore, if $\omega^{\prime}\in\{\sup_{\alpha\in[0,1]}|\widetilde{U}^{{\dagger}}(\alpha)-\alpha|<\sqrt{\log n/n}\}$, we have,

[TABLE]

and similarly,

[TABLE]

Thus there exists a constant $c_{1}$ such that for $n$ large enough (independent of $\omega^{\prime}$), $\widetilde{U}^{{\dagger}}(\varepsilon)\geq\varepsilon-c_{1}>0$ and $\widetilde{U}^{{\dagger}}(1-\varepsilon)\leq 1-\varepsilon+c_{1}<1$. Now as $\alpha\in[\varepsilon,1-\varepsilon]$, this implies that for $n$ large enough, $\xi\in[\varepsilon-c_{1},1-\varepsilon+c_{1}]$. Eventually, for $n$ large enough, $\omega^{\prime}\in\{\xi\in[\varepsilon-c_{1},1-\varepsilon+c_{1}]\}$. Let $\mathcal{Z}_{1}=\{\xi\notin[\varepsilon-c_{1},1-\varepsilon+c_{1}]\}$. We thus have for $n$ large enough, $\mathbb{P}^{*}(\mathcal{Z}_{1})\leq\mathbb{P}^{*}(\sup_{\alpha\in[0,1]}|u_{n}^{*}(\alpha)|\geq\sqrt{\log n})\leq 2/n^{2}$.

Now, from Lemma 14 and Lemma 15 there exist positive constants $c_{2}$ and $C_{3}$ such that for $n$ high enough,

[TABLE]

where $c_{2}$ is such that

[TABLE]

Let $c^{*}=c+C_{3}$ and let $\alpha_{1},\alpha_{2}\in[\varepsilon-c_{1},1-\varepsilon+c_{1}]$. Following the proof of Lemma 1.4.1 in (Csörgő, 1983), we have for $n$ high enough,

[TABLE]

Let $z_{4}\geq 1$ et let $\mathcal{Z}_{2}$ be the following event

[TABLE]

Let $\omega^{\prime}\in\overline{\mathcal{Z}_{1}}\cap\overline{\mathcal{Z}_{2}}$. We therefore have

[TABLE]

Therefore $\omega^{\prime}\in\overline{\mathcal{Z}_{3}}$ defined as the complement of

[TABLE]

Thus $\mathbb{P}^{*}(\overline{\mathcal{Z}_{1}}\cap\overline{\mathcal{Z}_{2}})\leq\mathbb{P}^{*}(\overline{\mathcal{Z}_{3}})$ which implies that for $n$ large enough,

[TABLE]

Now from the proof of Theorem 1.4.3 in (Csörgő, 1983) we can bound $\mathbb{P}^{*}(\mathcal{S}_{2})$ and obtain the result of the lemma. ∎