Overlap Coefficients Based on Kullback-Leibler Divergence: Exponential Populations Case

TL;DR

This paper introduces a new overlap coefficient based on Kullback-Leibler divergence for exponential populations, compares it with existing measures, and discusses statistical inference methods including confidence intervals and estimator properties.

Contribution

A novel overlap measure $ ext{ extLambda}$ based on Kullback-Leibler divergence is proposed for exponential populations, along with inference techniques and property analyses.

Findings

The new measure $ ext{ extLambda}$ is invariant and effective.

Confidence intervals for overlap measures are constructed using Taylor series.

Simulation studies evaluate bias and mean square error of estimators.

Abstract

This article is devoted to the study of overlap measures of densities of two exponential populations. Various Overlapping Coefficients, namely: Matusita's measure $\rho$, Morisita's measure $\lambda$ and Weitzman's measure $\Delta$. A new overlap measure $\Lambda$ based on Kullback-Leibler measure is proposed. The invariance property and a method of statistical inference of these coefficients also are presented. Taylor series approximation are used to construct confidence intervals for the overlap measures. The bias and mean square error properties of the estimators are studied through a simulation study.

Tables

OVL		lower limit $\left(L^{{}^{\prime }}\right)$		upper limit $\left(U^{{}^{\prime }}\right)$
$\Delta$		$1-L^{\frac{1}{1-L}}\left|1-\frac{1}{L}\right|$		$1-U^{\frac{1}{1-U}}\left|1-\frac{1}{U}\right|$
$\rho$		$\frac{2\sqrt{L}}{\left(L+1\right)}$		$\frac{2\sqrt{U}}{\left(U+1\right)}$
$\lambda$		$\frac{4L}{{\left(L+1\right)}^2}$		$\frac{4U}{{\left(U+1\right)}^2}$
$\Lambda$		$\frac{L}{L^2+L-1}$		$\frac{U}{U^2+U-1}$

	$\hat{\rho }$			$\hat{\lambda }$			$\hat{\Delta }$			$\hat{\Lambda }$
$n$	$Bias$	$MSE$	$Ratio$	$Bias$	$MSE$	$Ratio$	$Bias$	$MSE$	$Ratio$	$Bias$	$MSE$	$Ratio$
c=0.2	$\rho =0.745$			$\lambda =0.556$			$\Delta =0.465$			$\Lambda =0.24$
20	-0.029	0.007	-0.36	-0.030	0.016	-0.25	-0.0180	0.008	-0.061	0.0060	0.0080	0.067
50	-0.011	0.003	-0.22	-0.012	0.006	-0.15	-0.0070	0.007	-0.030	0.0020	0.0030	0.041
100	-0.055	0.001	-0.15	-0.056	0.003	-0.11	-0.0034	0.0015	-0.017	0.0011	0.0015	0.029
200	-0.003	0.000∗	-0.11	-0.003	0.001	-0.07	-0.0020	0.027	0.010	0.000∗	0.000∗	0.020
500	-0.001	0.000∗	-0.07	-0.001	0.000∗	-0.05	0.000∗	0.000∗	-0.039	0.000∗	0.000∗	0.013
c=0.5	$\rho =0.943$			$\lambda =0.889$			$\Delta =0.750$			$\Lambda =0.667$
20	-0.036	0.0040	-0.71	-0.640	0.0140	-0.66	-0.031	0.014	-0.092	0.048	0.0500	0.22
50	-0.014	0.0010	-0.44	-0.024	0.0040	-0.41	-0.012	0.005	-0.045	0.018	0.0190	0.013
100	-0.007	0.000∗	-0.31	-0.012	0.0020	-0.28	-0.006	0.0024	-0.026	0.009	0.0090	0.095
200	-0.003	0.000∗	-0.27	-0.006	0.000∗	-0.20	-0.003	0.001	-0.015	0.004	0.0045	0.067
500	-0.001	0.000∗	-0.13	-0.002	0.000∗	-0.13	-0.001	0.000∗	-0.05	-0.0018	0.0018	-0.042
c=0.8	$\rho =0.994$			$\lambda =0.988$			$\Delta =0.918$			$\Lambda =0.952$
20	-0.032	0.001	-0.87	-0.063	0.005	-0.87	-0.037	0.016	-0.3	-0.20	0.061	-0.84
50	-0.012	0.000∗	-0.74	-0.024	0.0011	-0.73	-0.014	0.006	-0.19	-0.079	0.013	-0.69
100	-0.006	0.000∗	-0.61	-0.012	0.000∗	-0.6	-0.007	0.0027	-0.133	-0.039	0.005	-0.56
200	-0.003	0.000∗	-0.47	-0.006	0.000∗	-0.47	-0.003	0.001	-0.09	-0.019	0.002	-0.43
500	-0.001	0.000∗	-0.32	-0.002	0.000∗	-0.32	-0.001	0.000∗	-0.06	-0.008	0.000∗	-0.28

Equations

$$\Delta =\int \min \left[f_1(x),f_2(x)\right]dx.$$

$$\rho =\int \sqrt{f_1(x)f_2(x)}dx$$

$$\lambda =\frac{2\int f_1(x)f_2(x)dx}{\int [f_1(x){]}^{2}dx+\int [f_2(x){]}^{2}dx}$$

$$\Lambda =\frac{1}{1+KL(f_1\Vert f_2)}$$

$$f_i(x;{\theta }_i)={\theta }_i\exp ({\theta }_{i}x),forx\in (0,\infty ).$$

$$\Delta =1-|1-\frac{1}{R}|R^{\frac{1}{1-R}}R\ne 1$$

$$\rho =\frac{2\sqrt{R}}{1+R}$$

$$\lambda =\frac{4R}{(1+R{)}^2}$$

$$\Lambda =\frac{R}{R^2-R+1}$$

$$f_1(x)=\frac{1}{{\theta }_1}\exp \{-\frac{x}{{\theta }_1}\}x>0$$

$$f_2(x)=\frac{1}{{\theta }_2}\exp \{-\frac{x}{{\theta }_2}\}x>0$$

$${\hat{\theta }}_1={\overline{X}}_1=\frac{{\sum }_{i=1}^{n_1}{X}_{1i}}{n_1}$$

$${\hat{\theta }}_2={\overline{X}}_2=\frac{{\sum }_{i=1}^{n_2}{X}_{2i}}{n_2}$$

$${\hat{\theta }}_1\sim G(n_1,\frac{{\theta }_1}{n_1}){\hat{\theta }}_2\sim G(n_2,\frac{{\theta }_2}{n_2})$$

$$Var({\hat{R}}^{\ast })=R^2\frac{(n_1+n_2-1)}{n_1(n_2-2)}$$

$$\hat{\Delta }=1-|1-\frac{1}{{\hat{R}}^{\ast }}|({\hat{R}}^{\ast }{)}^{\frac{1}{1-{\hat{R}}^{\ast }}}$$

$$\hat{\rho }=\frac{2\sqrt{{\hat{R}}^{\ast }}}{1+{\hat{R}}^{\ast }}$$

$$\hat{\lambda }=\frac{4{\hat{R}}^{\ast }}{(1+{\hat{R}}^{\ast }{)}^2}$$

$$\hat{\Lambda }=\frac{\hat{R}}{{\hat{R}}^2-\hat{R}+1}$$

$$Var(\hat{\Delta })=\frac{(n_1+n_2-1)(R{)}^{\frac{2}{1-R}}(\log R{)}^2}{n_1(n_2-2)(1-R{)}^2}$$

$$Var(\hat{\rho })=\frac{R(1-R{)}^2(n_1+n_2-1)}{n_1(n_2-2)(1+R{)}^4}$$

$$Var(\hat{\lambda })=\frac{16R^2(1-R{)}^2(n_1+n_2-1)}{n_1(n_2-2)(1+R{)}^6}$$

$$Var(\hat{\Lambda })=\frac{(n_1+n_2-1)}{n_1(n_2-2)}\frac{R^2(1-R^2{)}^2}{(R^2-R+1{)}^4}$$

$$\displaystyle Bias(\widehat{\Delta})=\displaystyle\left\{\begin{array}[]{l}\frac{(n_{1}+n_{2}-1)R^{2}}{n_{1}(n_{2}-2)}\frac{R^{\frac{2R-1}{1-R}}[R(2R-\log(R)-2)\log(R)-(R-1)^{2}]}{(R-1)^{3}}\quad if\quad R>1\\ \\ \frac{(n_{1}+n_{2}-1)R^{2}}{n_{1}(n_{2}-2)}\frac{R^{\frac{2R-1}{1-R}}[R(2R-\log(R)-2)\log(R)-(R-1)^{2}]}{(1-R)^{3}}\quad if\quad R<1\end{array}\right.$$

$$Bias({\hat{\rho }}^{\ast })=\frac{(n_1+n_2-1)\sqrt{R}}{n_1(n_2-2)}\frac{3R(R-2)-1}{2(R+1{)}^3}$$

$$Bias({\hat{\lambda }}^{\ast })=\frac{n_1+n_2-1}{n_1(n_2-2)}\frac{8R^2(R-2)}{(R+1{)}^4}$$

$$Bias(\hat{\Lambda })=-\frac{n_1+n_2-1}{n_1(n_2-2)}\frac{R^2(2R^3-6R+2)}{(R^2-R+1{)}^3}$$

$$\mathbb{P}\left(F_{(2n_1,2n_2)}^{\alpha /2}<\frac{{\theta }_2}{{\theta }_1}\hat{R}<F_{(2n_1,2n_2)}^{1-\alpha /2}\right)=1-\alpha$$

$$L=\frac{\hat{R}}{F_{(2n_1,2n_2)}^{1-\alpha /2}}andU=\frac{\hat{R}}{F_{(2n_1,2n_2)}^{\alpha /2}}$$

Full text

††journal: Journal Name

1 Introduction

The similarity between two densities can be considered as the commonality shared by both populations. Generally it is measured on the scale of [math] to $1$. Values of measure close to [math] corresponding to the distributions having supports with no intersection and $1$ to the perfect matching of the two distributions. Scientists from different disciplines propose different measures of similarity serving different purposes.

By using delta method Smith [20] derived formulas for estimating the mean and the variance of the discrete version of Weizman’s measure (also known as the overlap coefficient). Mishra et al. [12] gave the small and large sample properties of the sampling distributions for a function of this overlap measure estimator, under the assumption of homogeneity of variances for the case of two normal distributions. Mulekar and Mishra [14] simulated the sampling distribution of estimators of the overlap measures when the two densities correspond to the normal case with equal means and obtained the approximate expressions for the bias and variance of their estimators.

Smith [20] derived approximate formulas using the delta method for estimating the mean and variance of the discrete version of one such measure known as Weitzman’s measure $\Delta$(Weitzman [21]) (also known as the overlap coefficient). Mishra et al. [12] gave some properties of the sampling distributions for a function of the estimator, under the assumption of homogeneity of variances for the case of two normal distributions. Recently, several authors including Bradley and Piantadosi [4], Inman and Bradley [8], Clemons [5], Reiser and Faraggi [18], Clemons and Bradley [6], Mulekar and Mishra [15], Al-Saidy, et al. [1], Al-Saleh and Samawi [2], and Samawi and Al-Saleh [19] considered this measure.

Dixon [7] described the use of bootstrap and jackknife techniques for the Gini coefficient of size hierarchy, a commonly used measure of similarity between income distributions of two ethnic, gender, or geographical groups, and the Jaccard index of community similarity. AL-Saidy et al. [1] consider the problem of drawing inference about the three overlap measures under the Weibul distribution function with equal shape parameter. Wei Ning et al [16] have compared mixtures of generalized lambda distributions (GLDs) with normal mixtures by using KullbackLeibler $(KL)$ distance and overlapping coefficient $(\delta)$ .

The main objective of this paper is to propose a new $OVL$ based on the Kulback-Leibler divergence [9] for two Exponential distributions, i.e. from a measure of divergence or dissimilarity, we construct a measure of similarity noted $\Lambda$ defined in (1). We provide its maximum likelihood estimator.

The coefficients and their properties are given in section 2. The expressions for approximate bias and variance of $OVL$ are included in section 3. A method for making statistical inferences about the $OVLs$ is also discussed in this section. The results of simulation study are described in section 4, along with an example demonstrating the usefulness of $OVLs$. Finally, the conclusion and perspective is presented in Section 5.

2 Overlap Coefficients

We consider four different similarity measures (the overlap coefficients ($OVL$)): Matusita’s measure $\rho$, Morisita’s measure $\lambda$, Weitzman’s measure $\Delta$ and the measure based Kullback-Leibler divergence $\Lambda$. The overlap measure ($OVL$) is defined as the area of intersection of the graphs of two probability density functions. It measures the similarity, which is the agreement or the closeness of the two probability distributions.

Let $F_{1}(x)$ and $F_{2}(x)$ be two distribution functions with the corresponding density functions with respect to the Lebesgue measure. Four commonly used measures that describe the closeness between $F_{1}(x)$ and $F_{2}(x)$ are described below;

Weitzman’s Measure [21] The overlapping coefficient $\Delta$ is the area under two functions simultaneously, defined as,

[TABLE]

• 2.

Matusita’s Measure [11] second measure studied here is known as the Matusita’s measure, $\rho$, which is defined as,

[TABLE]

This measure is based on the distance between two functions (Matusita [11]). Matusita actually developed a discrete version of $\rho$, which is also known as the Freeman-Tukey measure (FT). This measure is related to the Hellinger distance (Rao [17] and Beran [3]).

• 3.

Morisita’s Measure [13] Morisita proposed an index of similarity between communities. Consider an ecological study involving two populations from each of which a random sample is taken, defined as,

[TABLE]

• 4.

Kullback-Leibler [9] : The Kullback-Leibler divergence was originally introduced by Solomon Kullback and Richard Leibler in 1951 as the directed divergence between two distributions. It is discussed in Kullback’s historic text, Information Theory and Statistics.

the overlap coefficient $\Lambda$ is the complement of Kullback-Leibler

[TABLE]

with $KL(f_{1}\|f_{2})=\int(f_{1}-f_{2})\log\left(\frac{f_{1}}{f_{2}}\right)dx$

2.1 Overlap measures (OVL) for Exponential Distribution

The simplest and most commonly used distribution in survival and reliability analysis is the one-parameter exponential distribution. Suppose $f_{i}(x;\theta_{i})$ indicate two exponential populations with respective hazard rates $\theta_{i}>0(i=1,2)$, that is

[TABLE]

The Overlapping Coefficients is shown graphically in Figure 2.

Let $R=\frac{\theta_{1}}{\theta_{2}}$, the ratio of hazard rates, then these measures can be shown to be functions of $R$ as follows

[TABLE]

[TABLE]

[TABLE]

and

[TABLE]

Lemma 1

For OVLs defined earlier,

• a)

$0\leq OVL\leq 1$* for all $R\geq 0$*

• b)

$OVL=1$* iff $R=1$*

• c)

$OVL=0$* iff $R=0$ or $R=\infty$*

ll four OVLs possess properties of reciprocity, invariance, and piecewise monotonicity

• a)

$OVL(R)=OVL(1/R)$**

• b)

$OVLs$* are monotonically increasing in $R$ for $0\leq R\leq 1$ and decreasing in $R>1$*

3 Bias and Variance of Estimates

As noted earlier, the overlap coefficients are functions of the ratio. Most commonly, in the estimation of ratios, estimators that are convenient and easy to understand are found to be biased. As noted by Lu, et al. (1989), the OVLs in this study are no exception to it. The amount of bias is $B(OVL)=\mathbb{E}(OVL)-OVL$. To examine the effects of bias, approximate expressions for the mean and the variance of estimates are obtained.

suppose that $(X_{ij};j=1,...,n_{i};i=1,2)$ denote independent observation from two independent random samples draw from $f_{1}(x)$ and $f_{2}(x)$ respectively, where

[TABLE]

and

[TABLE]

The maximum likelihood estimators (MLEs) based on the two samples are given by:

From the first sample:

[TABLE]

• 2)

From the second sample:

[TABLE]

Note that, it is easy to show that

[TABLE]

where $G(.,.)$ stands for the gamma distribution function. Hence, the variances of those MLE’s are respectively $Var(\widehat{\theta}_{1})=\frac{\theta_{1}^{2}}{n_{1}}$ and $Var(\widehat{\theta}_{2})=\frac{\theta_{2}^{2}}{n_{2}}$ Then we may define an estimate of $R$ is $\widehat{R}=\frac{\widehat{\theta}_{1}}{\widehat{\theta}_{2}}$.

Therefore, using the relationship between Gamma distribution and Chi-square distribution and the fact that the two samples are independent, it is easy to show that $\frac{\theta_{1}}{\theta_{2}}\widehat{R}$ has $F$-distribution (i.e, $F(2n_{1},2n_{2})$). Hence, the variance of $\widehat{R}$ is $Var(\widehat{R})=R^{2}\frac{n_{2}^{2}(n_{1}+n_{2}-1)}{n_{1}(n_{2}-1)(n_{2}-2)}$ Also, an unbiased estimate of $R$ is given by $\widehat{R}^{*}=\frac{\widehat{\theta}_{1}}{\widehat{\theta}_{2}}\frac{(n_{2}-1)}{n_{2}}=\frac{(n_{2}-1)}{n_{2}}\widehat{R}$ with

[TABLE]

. Clearly, $\widehat{R}^{*}$ has less variance than $\widehat{R}$.

[TABLE]

[TABLE]

[TABLE]

and

[TABLE]

Theorem 1

Suppose $\widehat{\Delta}$, $\widehat{\rho}$, $\widehat{\lambda}$ and $\widehat{\Lambda}$ are the estimates of $\Delta$, $\rho$, $\lambda$ and $\Lambda$ respectively, obtained replacing $R$ by $\widehat{R}^{*}$. the approximate sampling variance of the $OVL$ measures can be obtained as follows:

[TABLE]

[TABLE]

[TABLE]

[TABLE]

Proof 1

Since each of the $OVL$ is a function of $R$, the expressions are obtained using the first order Taylor series expansion about $R$ and the $Var(\widehat{R}^{*})$ given in equation (2).

Theorem 2

the approximate sampling bias of the$OVL$ measures can be obtained as follows:

[TABLE]

[TABLE]

[TABLE]

[TABLE]

Proof 2

Using the second order Taylor series expansion the desired results are obtained.

Remark 1

Reasonable estimates for the above variances and the biases can be obtained by substituting $R$ by its consistency estimator $\widehat{R}^{*}$ in the above formulas.

4 Confidence Interval Eestimation of Overlap

From Section 3, $\frac{\widehat{R}}{R}\sim F(2n_{1},2n_{2})$, then $\frac{\theta_{2}n_{2}}{\theta_{1}(n_{2}-1)}\widehat{R}^{*}\sim F(2n_{1},2n_{2})$. Let $L$ and $U$ be the lower and upper confidence limits respectively of $R$, corresponding to the probability $1-\alpha$, i.e., $\mathbb{P}(L<R<U)=1-\alpha$. Thus $L$ and $U$ can be determined by solving for $R$ the equation

[TABLE]

where $F^{\alpha/2}_{(2n_{1},2n_{2})}$ and $F^{1-\alpha/2}_{(2n_{1},2n_{2})}$ are the lower and the upper $\alpha/2$ quantile of the $F(2n_{1},2n_{2})$ distribution respectively. Thus

[TABLE]

The lower $(L^{{}^{\prime}})$ and upper $(U^{{}^{\prime}})$ limits of OVLs can be obtained using appropriate transformation as $1-\alpha=Pr(L^{{}^{\prime}}<OVL(R)<U^{{}^{\prime}})$. Here $L^{{}^{\prime}}=OVL(L)$ and $U^{{}^{\prime}}=OVL(U)$. The confidence limits for OVLs are as follows:

If $(L,U)\in(1\infty)$, then the $L^{{}^{\prime}}$ and $U^{{}^{\prime}}$ interchange their role and the confidence interval for OVL becomes $(U^{{}^{\prime}},L^{{}^{\prime}})$ If 1 is enclosed in the interval $(L,U)$, then it asserts at $OVL=1$.

5 Simulation Study

A Monte Carlo study was conducted using to evaluate the performance of approximations to bias and variance of four overlap coefficients. From each population $1000$ samples of $20,50,100$, $200$ and $500$ observations were generated. $\widehat{\rho}$, $\widehat{\lambda}$, $\widehat{\Delta}$ and $\widehat{\Lambda}$ were computed for each pair of samples. The bias and variance of estimates were computed using actual OVLs and the estimates. The bias and MSE for $R=0.2,0.5,0.8$ are reported in Table 1.

The following conclusions are drawn based on these computations where only the values of $R<1$ are considered. However, for the Overlap measures, the case $R<1$ is symmetric to the case $R>1$ the comments given below in terms of $R$ can also be interpreted in terms of $1/R$ for these OVL measures.

For sample sizes larger than 50, the bias is fairly close to zero. Weitzman’s measure has less bias than others but Morisita’s measure has the largest bias.

The bias decreases as sample size increases, as expected and the MSE goes to zero for each OVLs. $\Lambda$ tend to be more biased and the sampling distributions show larger variability.

It is clear that the actual OVLs are found to be underestimated (Figure $3$) and for very small values of $R$ and small sample sizes, they are observe to be overestimated. The bias approaches [math] very fast. For $n\geq 50$, the amount of bias is negligible and fairly close to 0. Although $\widehat{\Lambda}$ has less bias than the other in case $R=0.2$ and has the largest bias for $R=0.8$; the bias of Delta approaches [math] faster than the other three. The bias of $\widehat{\lambda}$ is the slowest in approaching [math].

An important increase in standard deviations for small values of $R$ is observed for $\rho$ and $\lambda$. For $\Delta$ standard deviation increases as $R$ approaches $1$. But a remarkable increase in standard deviations for moderate values of $R$ in the $\Lambda$ case (Figure $4$). They decrease fast as $n$ increases, from $n=100$ the standard deviations are negligible. The difference between the $MSE$ of $\rho$ and $\Lambda$ is almost nil for small values of $R$, but the difference increases as $R$ becomes large with $\rho$ giving lowest $MSE$ values and $\Lambda$ the highest.

The estimates of MSE are plotted in Figure 5 for all four overlap coefficients. As the sample size increases, the MSE reduces considerably.

6 Conclusion

The problem of estimation of four commonly used measures of overlap for two exponential densities with heterogeneous variances is considered and relations between them are studied. Overlap coefficients are used frequently to describe the degree of interspecific encounter or crowdedness of two species in their resource utilization.

Relations between three commonly used measures of overlap with our measure of overlap are studied and approximate expressions for the bias and the variance of the estimates are presented. The invariance property and a method of statistical inference of these coefficients also are presented. Monte Carlo evaluations are used to study the bias and precision of the proposed overlap measures.