On discrimination between two close distribution tails

TL;DR

This paper introduces a new goodness-of-fit test based on higher order statistics to distinguish between two similar distribution tails, proving its consistency without assuming maximum domain of attraction.

Contribution

It proposes a novel tail discrimination test that is consistent under various conditions, expanding applicability beyond traditional assumptions.

Findings

Test is consistent for different alternatives

Does not require maximum domain of attraction assumption

Applicable to distinguishing close distribution tails

Abstract

The goodness-of-fit test for discrimination of two tail distribution using higher order statistics is proposed. The consistency of proposed test is proved for two different alternatives. We do not assume belonging the corresponding distribution function to a maximum domain of attraction.

Equations

$$\frac{1-G(x)}{(1-F(x){)}^{1-\epsilon }}\text{ is nondecreasing with}x>x_0.$$

$$\epsilon (F,G)=\max \{\epsilon :F,G\text{ satisfy either }B(F,G)\text{ or }B(G,F)\text{ for }\epsilon \}.$$

$$R_{k,n}=\ln (1-F_0(X_{(n-k)}))-\frac{1}{k}\sum _{i=n-k+1}^n\ln (1-F_0(X_{(i)})).$$

$$R_{k,n}\stackrel{d}{=}{\gamma }_H/\gamma ,$$

$$\sqrt{k}(R_{k,n}-1)\stackrel{d}{⟶}\xi \text{ as }k,n\to \infty ,$$

$$\sqrt{k_n}|R_{k_n,n}-1|\stackrel{d}{⟶}+\infty$$

$$\underset{F_1\in {\Theta }_{\epsilon }(F_0)}{\inf }\sqrt{k_n}|R_{k_n,n}-1|\stackrel{d}{⟶}+\infty .$$

$$\frac{1-G(x)}{(1-F(x))(-\ln (1-F(x)){)}^{\epsilon }}\text{ is nondecreasing, }x>x_0.$$

$$1-F_1(x)\le (1-F_0(x){)}^{\delta },x>x_0.$$

$${\epsilon }^{\prime }(F,G)=\max \{\epsilon :F,G\text{ satisfy either }C(F,G)\text{ or }C(G,F)\text{ for }\epsilon \}.$$

$$\sqrt{k_n}|R_{k_n,n}-1|\stackrel{d}{⟶}+\infty$$

$$\underset{F_1\in {\Theta }_{\epsilon }^{\prime }(F_0)}{\inf }\sqrt{k_n}|R_{k_n,n}-1|\stackrel{d}{⟶}+\infty .$$

$$F_q(x)=P(X\le x|X>q)=\frac{F(x)-F(q)}{1-F(q)},x>q.$$

$${\eta }_q=\ln \left(\frac{1-F(q)}{1-F({\xi }_q)}\right).$$

$$P({\eta }_q\le y)=P\left(\ln \left(\frac{1-F(q)}{1-F({\xi }_q)}\right)\le y\right)=P\left(\frac{1-F(q)}{1-F({\xi }_q)}\le e^y\right)=$$

$$=P\left(F({\xi }_q)\le 1-(1-F(q))e^{-y}\right)=P\left({\xi }_q\le F^{\leftarrow }\left(1-\frac{1-F(q)}{e^y}\right)\right).$$

$$P\left({\xi }_q\le F^{\leftarrow }\left(1-\frac{1-F(q)}{e^y}\right)\right)=\frac{F\left(F^{\leftarrow }\left(1-\frac{1-F(q)}{e^y}\right)\right)-F(q)}{1-F(q)}=1-e^{-y}.$$

$$P({\eta }_q\le y)=\frac{G\left(F^{\leftarrow }\left(1-\frac{1-F(q)}{e^y}\right)\right)-G(q)}{1-G(q)}\ge$$

$$\frac{F\left(F^{\leftarrow }\left(1-\frac{1-F(q)}{e^y}\right)\right)-F(q)}{1-F(q)}=1-e^{-y}.$$

$$\frac{G\left(F^{\leftarrow }\left(1-\frac{1-F(q)}{e^y}\right)\right)-G(q)}{1-G(q)}\ge 1-e^{-y}⟺\frac{1-G\left(F^{\leftarrow }\left(1-\frac{1-F(q)}{e^y}\right)\right)}{1-G(q)}\le e^{-y}$$

$$⟺G\left(F^{\leftarrow }\left(1-\frac{1-F(q)}{e^y}\right)\right)\le 1-\frac{1-G(q)}{e^y}⟺$$

$$F^{\leftarrow }\left(1-\frac{1-F(q)}{e^y}\right)\le G^{\leftarrow }\left(1-\frac{1-G(q)}{e^y}\right).$$

$$e^{-y}=\frac{1-G(z_G)}{1-G(q)}=\frac{1-F(z_F)}{1-F(q)}.$$

$$\frac{1-F(z_F)}{1-F(q)}=\frac{1-G(z_G)}{1-G(q)}\le \frac{1-G(z_F)}{1-G(q)}.$$

$$\frac{G(x)-G(q)}{1-G(q)}\ge \frac{F(x)-F(q)}{1-F(q)}\forall x>q\ge x_0⟺\frac{1-G(x)}{1-G(q)}\le \frac{1-F(x)}{1-F(q)}\forall x>q\ge x_0⟺$$

$$\frac{1-G(x)}{1-F(x)}\le \frac{1-G(q)}{1-F(q)}\forall x>q\ge x_0⟺\frac{1-G(x)}{1-F(x)}\text{ is nonincreasing for all }x>x_0.■$$

$${\left\{-\ln (1-F_0(X_{(n-i)}))+\ln (1-F_0(X_{(n-k)}))\right\}}_{i=0}^{k-1}\stackrel{d}{=}{\left\{\sum _{j=i+1}^k\frac{E_{n-j+1}}{j}\right\}}_{i=0}^{k-1},$$

$${\left\{-\ln (1-F_0(X_{(n-i)}))+\ln (1-F_0(X_{(n-k)}))\right\}}_{i=0}^{k-1}\stackrel{d}{=}{\left\{E_{(k-i)}\right\}}_{i=0}^{k-1},$$

$$\sqrt{k}(R_{k,n}-1)\stackrel{d}{=}\sqrt{k}\left(\frac{1}{k}\sum _{i=0}^{k-1}{E}_{(k-i)}-1\right)=\sqrt{k}\left(\frac{1}{k}\sum _{j=1}^k{E}_j-1\right),$$

$$Y_i=\ln (1-F_0(q))-\ln (1-F_0(X_i^{\ast })),$$

Full text

On discrimination between two close distribution tails.

Rodionov I. V Moscow State University, Faculty of Mathematics and Mechanics and Moscow Institute of Physics and Technology, Faculty of Innovations and High Technologies. E-mail: vecsell@gmail.com

1 Introduction. Main result.

Statistics deals often with discrimination of close distributions based on censored or truncated data, in particular, for high-risk insurances and reliability problems. The situation when one observes data exceeding a pre-determined threshold is well-studied, see [1], [2], [3] and references therein. On the other hand statistics of extremes says that only higher order statistics should be used for discrimination of close distribution tails, wherein moderate sample values can be modeled with standard statistical tools. In particular, such approach for distributions from Gumbel maximum domain of attraction (for the definitions see [4]) is considered in [5], [6], [7]. As well, any estimators of the extreme value indices $\gamma$ and $\rho$ (see [8]) can be used also to discriminate the distribution tails. Notice that we do not assume belonging the corresponding distribution function to a maximum domain of attraction.

Definition 1

The distribution functions $F$ and $G$ are said to be satisfied the condition $B(F,G)$ if for some $\varepsilon>0$ and $x_{0}$

[TABLE]

Denote by $\Theta(F_{0})$ the class of continuous distribution functions $F_{1}$ satisfying either $B(F_{1},F_{0})$ or $B(F_{0},F_{1}).$ Consider the simple hypothesis $H_{0}:F=F_{0}$ and the alternative hypothesis $H_{1}:F\in\Theta(F_{0}),$ where $F_{0}$ is continuous. Notice that if distribution functions $F,$ $G$ satisfy either $B(F,G)$ or $B(G,F)$ for some $\varepsilon>0$ then it holds for all $\varepsilon_{1},\ 0<\varepsilon_{1}<\varepsilon.$ So denote

[TABLE]

Denote by $\Theta_{\varepsilon}(F_{0})$ the class of continuous distribution functions $F_{1}$ satisfying either $B(F_{1},F_{0})$ or $B(F_{0},F_{1})$ with $\varepsilon(F_{0},F_{1})\geq\varepsilon$ and consider another alternative hypothesis $H_{1,\varepsilon}:F\in\Theta_{\varepsilon}(F_{0}).$ Let $X_{1},\ldots,X_{n}$ be i.i.d. random variables with a common distribution function $F$. Denote by $X_{(1)}\leq\ldots\leq X_{(n)}$ the order statistics for them. Introduce the Hill-like statistics

[TABLE]

which we are going to use for the problem of discrimination between the two introduced above hypotheses when $k$ higher order statistics are known. Remark that if $F_{0}$ is Pareto distribution function with parameter $\gamma$, then

[TABLE]

where $\gamma_{H}$ is the Hill estimator of $\gamma.$ If furthermore $F_{0}$ belongs to Fréchet max-domain of attraction, then $R_{k,n}$ behaves asymptotically as $\gamma_{H}/\gamma,$ that is, theirs ratio tends to one as $n\rightarrow\infty.$ We will show that the distributions of $R_{k,n}$ if either $H_{0}$ or $H_{1}$ fulfilled are different which can give a statistical for discrimination the hypotheses. The following two results describe the behavior of $R_{k,n}$ as $k,n\rightarrow\infty$ with $k<n$ provided $H_{0}$ or $H_{1}$ is fulfilled.

Theorem 1

If $H_{0}$ holds then

[TABLE]

where $\xi$ is standard normal random variable, i.e. $\xi\sim N(0,1).$

This theorem gives obvious goodness-of-fit test for the tail of $F.$ Besides, the following result provides some information about the consistency of this test. Assume that $H_{0}$ does not hold and $F$ is equal to $F_{1}$ which is different from $F_{0}.$ Denote $x^{\ast}$, the right endpoint of $F_{1}$, that is, $x^{\ast}=\sup\{x:F_{1}(x)<1\}.$ Assume that $F_{0}$ and any $F_{1}\in\Theta(F_{0})$ have the same right endpoint (how to discriminate distributions with different endpoints, see [10], [4]). Further consider $x^{\ast}=+\infty,$ otherwise change variables $y=1/(x^{\ast}-x)$ gives the assumption. The following theorem shows consistency of the proposed test.

Theorem 2

1. (i)

If $H_{1}$ holds then

[TABLE]

provided $k_{n}\to\infty,$ $k_{n}/n\to 0$ as $n\to\infty.$ 2. (ii)

If $H_{1,\varepsilon}$ holds then under the same conditions

[TABLE]

The considered test makes it possible to discriminate, for example, two normal distributions with different variances, but we should weaken the condition (1) to discriminate two normal distributions with the same variance and different means. But weakening the condition (1) imposes some conditions on behavior of the sequence $k_{n}.$

Definition 2

The distribution functions $F$ and $G$ are said to satisfy the condition $C(F,G)$ if for some $\varepsilon>0$ and $x_{0}$

[TABLE]

Denote by $\Theta^{\prime}(F_{0})$ the class of continuous distribution functions $F_{1}$ satisfying either $C(F_{1},F_{0})$ or $C(F_{0},F_{1})$ and the following condition: for some $\delta\in(0,1)$

[TABLE]

See, if distribution functions $F,$ $G$ satisfy either $C(F,G)$ or $C(G,F)$ for some $\varepsilon>0$ then it holds for all $\varepsilon_{1},\ 0<\varepsilon_{1}<\varepsilon.$ Denote

[TABLE]

Denote by $\Theta_{\varepsilon}^{\prime}(F_{0})$ the class of continuous distribution functions $F_{1}$ satisfying (3) and either $C(F_{1},F_{0})$ or $C(F_{0},F_{1})$ with $\varepsilon^{\prime}(F_{0},F_{1})\geq\varepsilon.$ As before, consider the simple hypothesis $H_{0}:F=F_{0}$ and two alternative hypotheses $H_{1}^{\prime}:F\in\Theta^{\prime}(F_{0}),$ $H_{1,\varepsilon}^{\prime}:F\in\Theta^{\prime}_{\varepsilon}(F_{0})$ with continuous $F_{0}.$

Theorem 3

1. (i)

If $H_{1}^{\prime}$ holds then

[TABLE]

provided $k_{n}/n\to 0,$ $k_{n}^{1/2-\alpha}/\ln n\to+\infty,$ for some $\alpha\in(0,1/2),$ as $n\to\infty.$ 2. (ii)

If $H_{1,\varepsilon}^{\prime}$ holds then under the same conditions

[TABLE]

2 Auxiliary results and proofs.

2.1 Auxiliary results.

Since $R_{n}$ depends on the higher order statistics we cannot immediately use independence of the random variables $(X_{1},\ldots,X_{n}).$ Therefore consider the conditional distribution of $R_{n}$ given $X_{(n-k)}=q$ applying the following lemma.

Lemma 1

([4]) Let $X,X_{1},\ldots,X_{n}$ be i.i.d. random variables with common distribution function $F,$ and let $X_{(1)}\leq\ldots\leq X_{(n)}$ be the $n$th order statistics. For any $k=1\ldots n-1$, the conditional joint distribution of $\{X_{(i)}\}_{i=n-k+1}^{n}$ given $X_{(n-k)}=q$ is equal to the (unconditional) joint distribution of the corresponding set $\{X_{(i)}^{\ast}\}_{i=1}^{k}$ of order statistics for i.i.d. random variables $\{X_{i}^{\ast}\}_{i=1}^{k}$ having the distribution function

[TABLE]

We call $F_{q}(x),$ $x>q,$ the tail distribution function linked with the distribution function $F.$ Consider two continuous distribution functions $F$ and $G$ and a random variable $\xi_{q}$ with distribution function $G_{q},$ where $q\in\mathbb{R}$ is some parameter. Let

[TABLE]

Clear, $\eta_{q}\geq 0$ for all $q\in\mathbb{R}.$

The crucial point in the proof of Theorem 2 is studying of asymptotical behavior of $\eta_{q}.$

Proposition 1

Let $F_{q}$ and $G_{q}$ are tail distribution functions of $F$ and $G$ respectively. Then

(i)

If for some $x_{0}$, $q>x_{0},$ and any $x>q$, $F_{q}(x)=G_{q}(x),$ then $\eta_{q}$ is standard exponential.

(ii)

$G_{q}(x)\geq F_{q}(x)$* for any $x>q$ if and only if $\eta_{q}$ is stochastically smaller than a standard exponential random variable.*

$G_{q}(x)\leq F_{q}(x)$* for any $x>q$ if and only if $\eta_{q}$ is stochastically larger than a standard exponential random variable.*

(iii)

$G_{q}(x)\geq F_{q}(x)$* for any $x>q\geq x_{0}$ and some $x_{0}$ if and only if $(1-G(x))/(1-F(x))$ is nonincreasing function as $x>x_{0}.$*

2.2 Proof of Proposition 1.

(i) Let $F_{q}(x)=G_{q}(x)$ for all $x>q,$ then we have for the distribution function of $\eta_{q}$,

[TABLE]

[TABLE]

Furthermore, for the same $x$,

[TABLE]

(ii) Now assume that for all $x>q$ and some $q\in\mathbb{R}$, $G_{q}(x)\geq F_{q}(x).$ Then from (4), since $1-(1-F(q))e^{-y}\geq F(q)$ for all $y\geq 0$ it follows that

[TABLE]

[TABLE]

Conversely, assume that $\eta_{q}$ is stochastically smaller than a standard exponential random variable, that is, $P(\eta_{q}\leq y)\geq 1-e^{-y}$ for all $y\geq 0.$ With (4) we get that

[TABLE]

[TABLE]

[TABLE]

Denote $z_{F}=F^{\leftarrow}\left(1-e^{-y}(1-F(q))\right)$ and $z_{G}=G^{\leftarrow}\left(1-e^{-y}(1-G(q))\right).$ Since $F(z_{F})=1-e^{-y}(1-F(q))$ and $G(z_{G})=1-e^{-y}(1-G(q)),$ we have,

[TABLE]

Further, since $z_{F}\leq z_{G}$ then

[TABLE]

This observation completes the proof since $z_{F}\in[q,\infty).$ The proof of the second assertion is similar.

(iii) We have,

[TABLE]

[TABLE]

2.3 Proof of Theorem 1.

Under the conditions of Theorem 1, $F_{0}(X_{1})$ is uniformly distributed on $[0,1]$, that is, $F_{0}(X_{1})$ $\sim U[0,1],$ hence $-\ln(1-F_{0}(X))$ is standard exponential random variable. It follows from Rényi’s representation (see [4]), that

[TABLE]

where $E_{1},E_{2}\ldots$ are independent standard exponential variables. Therefore the distribution of the left-hand side does not depend on $n$ and

[TABLE]

where $E_{(1)}\leq\ldots\leq E_{(k)}$ are the $n$th order statistics of the sample $\{E_{i}\}_{i=1}^{k}.$ Finally we have,

[TABLE]

and the assertion follows from the Central Limit Theorem.

2.4 Proof of Theorem 2.

We first prove (i). The steps of the proof are similar to corresponding steps in [6] and [7]. Consider asymptotic behavior of $R_{k_{n},n}$ as $n\rightarrow\infty.$ Denote

[TABLE]

where $\{X_{i}^{\ast}\}_{i=1}^{k_{n}}$ are i.i.d. random variables introduced in Lemma 1 with the distribution function

[TABLE]

Taking $F=F_{0}$ and $G=F_{1}$ we have, $Y_{i}\overset{d}{=}\eta_{q},$ $i\in\{1,\ldots,k_{n}\}$. Notice that, in view of Lemma 1, the joint distribution of order statistics $\{Y_{(i)}\}_{i=1}^{k_{n}}$ of the sample $\{Y_{j}\}_{i=1}^{k_{n}}$ is equal to the joint conditional distribution of order statistics $\{Z_{(j)}\}_{j=1}^{k_{n}}$ of $\{Z_{j}\}_{j=1}^{k_{n}}$ given $X_{(n-k_{n})}=q,$ where

[TABLE]

Clear,

[TABLE]

So, the conditional distribution of $R_{k_{n},n}$ given $X_{(n-k)}=q$ is equal to the distribution of $\frac{1}{k_{n}}\sum_{i=1}^{k_{n}}Y_{i}.$ Further, distribution functions $F_{1}$ and $F_{0}$ satisfy $B(F_{0},F_{1})$ or $B(F_{1},F_{0}).$ First suppose that the condition $B(F_{0},F_{1})$ holds for some $\varepsilon>0$ and $x_{0}.$ Since $x^{\ast}=+\infty,$ $X_{(n-k_{n})}\rightarrow+\infty$ a.s., we may consider the case $q>x_{0}$ only. Proposition 1 (iii) implies, that

[TABLE]

With (5), we get that,

[TABLE]

[TABLE]

hence $Y_{1}$ is stochastically larger than a random variable $E\sim Exp(1-\varepsilon),$ write $Y_{1}\gg E.$ Further, let $E_{1},\ldots,E_{k_{n}}$ are i.i.d. random variables with distribution function $H(x)=1-e^{-(1-\varepsilon)x},$ then

[TABLE]

Since (6) holds for all $q>x_{0},$ and $X_{(n-k_{n})}\rightarrow+\infty$ a.s. as $n\to\infty$, we have under the conditions of Theorem 2, that

[TABLE]

It follows from Lindeberg-Feller theorem, that

[TABLE]

therefore

[TABLE]

Finally, with (7), we have,

[TABLE]

If the condition $B(F_{0},F_{1})$ holds, then

[TABLE]

and the proof is similar. The second assertion easily follows from (7) and (8).

2.5 Proof of Theorem 3.

Firstly we prove (i). Denote $\overline{F}(x)=1-F(x).$ In notation of the proof of Theorem 2, find the distribution of $Y_{1}.$ First assume that $C(F_{0},F_{1})$ holds. With (5) and Proposition 1 (iii) we have,

[TABLE]

[TABLE]

For $\varepsilon,c\in(0,1),$

[TABLE]

and $G(x)=1-e^{-x}(1+c\varepsilon-c\varepsilon e^{-x})$ is the distribution function. Hence,

[TABLE]

Further, let $\zeta,\zeta_{1},\ldots,\zeta_{k_{n}}$ be i.i.d. random variables with this distribution function. Therefore, like the proof of Theorem 2,

[TABLE]

Clear,

[TABLE]

so we have,

[TABLE]

Consider now the statistic $\sqrt{k_{n}}/\ln\overline{F_{0}}(X_{(n-k_{n})}).$ Denote $R_{i}=\overline{F_{1}}(X_{i}),$ $i=1,\ldots,n.$ Since $F_{1}$ is continuous, $R_{1},\ldots,R_{n}$ are i.i.d. standard uniform random variables and $R_{(k_{n})}=\overline{F_{1}}(X_{(n-k_{n})}).$ Theorem 2.2.1 [4] implies, that

[TABLE]

Using the delta method (see [11]) for the function $f(x)=-x/\ln x,$ we have

[TABLE]

since under the conditions of theorem

[TABLE]

Further,

[TABLE]

[TABLE]

and (11) implies that the first summand in the right hand side tends to [math] in probability. Therefore,

[TABLE]

and under the conditions of Theorem 3,

[TABLE]

On the other hand, from (3) it follows that

[TABLE]

as $n\rightarrow\infty.$ Further, it follows from the Law of large numbers for triangular arrays (see [9]), that for any $\epsilon>0$

[TABLE]

It means that the term in the left hand side is asymptotically smaller in probability than $k_{n}^{\epsilon}.$ Hence for any $q,$ given $X_{(n-k_{n})}=q$

[TABLE]

and finally,

[TABLE]

If the condition $C(F_{1},F_{0})$ holds, then

[TABLE]

and the proof is the same. The second assertion clearly follows from (9), (10) and (12).