Divergence measures estimation and its asymptotic normality theory in the discrete case

TL;DR

This paper develops the asymptotic theory for $\

Contribution

It introduces the asymptotic properties of $\

Findings

Asymptotic normality of divergence estimators established.

Validation of theoretical results through simulations.

Derivation of statistical tests based on divergence measures.

Abstract

In this paper we provide the asymptotic theory of the general of $\phi$-divergences measures, which include the most common divergence measures : Renyi and Tsallis families and the Kullback-Leibler measure. We are interested in divergence measures in the discrete case. One sided and two-sided statistical tests are derived as well as symmetrized estimators. Almost sure rates of convergence and asymptotic normality theorem are obtained in the general case, and next particularized for the Renyi and Tsallis families and for the Kullback-Leibler measure as well. Our theorical results are validated by simulations.

Figures

Equations

$$\mathcal{P}(\mathcal{X})=\left\{\text{p}=(p_c{)}_{c\in \mathcal{X}}:p_c\ge 0,\forall c\in \mathcal{X}\text{and}\sum _{c\in \mathcal{X}}{p}_c=1\right\}.$$

$$\mathcal{D}:$$

$$\mathbb{P}(X=c_j)=p_j,\text{and}\mathbb{P}(Y=c_j)=q_j,j\in \{1,\cdots ,r\}.$$

$${\mathcal{D}}_{L_2}(\text{p},\text{q})=\sum _{j=1}^r(p_j-q_j{)}^2.$$

$${\mathcal{D}}_{R,\alpha }(\text{p},\text{q})=\frac{1}{\alpha -1}\log \left(\sum _{j=1}^r{p}_j^{\alpha }{q}_j^{1-\alpha }\right).$$

$${\mathcal{D}}_{T,\alpha }(\text{p},\text{q})=\frac{1}{\alpha -1}\left(\sum _{j=1}^r{p}_j^{\alpha }{q}_j^{1-\alpha }-1\right);$$

$${\mathcal{D}}_{KL}(\text{p},\text{q})=\sum _{j=1}^r{p}_j\log (p_j/q_j).$$

$$p_j>0\text{and}{q}_j>0,\forall j\in D=\{1,2,\cdots ,r\}(\text{BD})$$

$${\mathcal{D}}^{(s)}(\text{p},\text{q})=\frac{\mathcal{D}(\text{p},\text{q})+\mathcal{D}(\text{q},\text{p})}{2}$$

$${\mathcal{S}}_{\alpha }(\text{p},\text{q})=\sum _{j\in D}{p}_j^{\alpha }{q}_j^{1-\alpha },\text{with}\alpha \ne 1,\alpha >0.$$

$$\mathbb{E}\left|{\mathcal{D}}_{R,\alpha }({\hat{\text{p}}}_n,{\hat{\text{q}}}_n)-{\mathcal{D}}_{R,\alpha }(\text{p},\text{q})\right|\le c\left(n^{-1/2}+n^{-\frac{3s}{2s+d}}\right)$$

$$\underset{n,m\to \infty }{\lim }\mathbb{E}{\left[{\mathcal{D}}_{T,\alpha }({\hat{\text{p}}}_n,{\hat{\text{q}}}_m)-{\mathcal{D}}_{T,\alpha }(\text{p},\text{q})\right]}^2=0$$

$$J(\text{p},\text{q})=\sum _{j\in D}\varphi (p_j,q_j),$$

$$p_j=\mathbb{P}(X=c_j)\text{and}{q}_j=\mathbb{P}(Y=c_j).$$

$${\hat{\text{p}}}_n=({\hat{p}}_n^c{)}_{c\in \mathcal{X}},\text{where}{\hat{p}}_n^{c_j}$$

$${\hat{\text{q}}}_m=({\hat{q}}_m^c{)}_{c\in \mathcal{X}},\text{where}{\hat{q}}_m^{c_j}=\frac{1}{m}\sum _{i=1}^m{1}_{c_j}(Y_i),$$

$$J(\text{p},\text{q})=\sum _{j\in D}\varphi (p_j,q_j)$$

$$J({\hat{\text{p}}}_n,\text{q})$$

$$J({\hat{\text{p}}}_n,{\hat{\text{q}}}_m)=\sum _{j\in D}\varphi ({\hat{p}}_n^{c_j},{\hat{q}}_m^{c_j}).$$

$$a_n$$

$$c_{n,m}=\max (a_n,b_m).$$

$${\varphi }_1^{(1)}(s,t)=\frac{\partial \varphi }{\partial s}(s,t),{\varphi }_2^{(1)}(s,t)=\frac{\partial \varphi }{\partial t}(s,t)$$

$${\varphi }_1^{(2)}(s,t)=\frac{{\partial }^2\varphi }{\partial s^2}(s,t),{\varphi }_2^{(2)}(s,t)=\frac{{\partial }^2\varphi }{\partial t^2}(s,t),{\varphi }_{1,2}^{(2)}(s,t)={\varphi }_{2,1}^{(2)}(s,t)=\frac{{\partial }^2\varphi }{\partial s\partial t}(s,t).$$

$$A_{1,p}$$

$$A_{3,q}$$

$$\underset{n\to +\infty }{\mathrm{limsup}}\frac{|J({\hat{\text{p}}}_n,\text{q})-J(\text{p},\text{q})|}{a_n}\le A_{1,p},\text{a.s.}$$

$$\underset{m\to +\infty }{\mathrm{limsup}}\frac{\left|J(\text{p},{\hat{\text{q}}}_m)-J(\text{p},\text{q})\right|}{b_m}\le A_{2,q},\text{a.s.}$$

$$\underset{(n,m)\to (+\infty ,+\infty )}{\mathrm{limsup}}\frac{\left|J({\hat{\text{p}}}_n,{\hat{\text{q}}}_m)-J(\text{p},\text{q})\right|}{c_{n,m}}\le A_{1,p}+A_{2,q}\text{ a.s.}$$

$$V_{1,p}=\sum _{j\in D}{p}_j(1-p_j)({\varphi }_1^{(1)}(p_j,q_j){)}^2-2\sum _{(i,j)\in D^2,i\ne j}{p}_i{p}_j{\varphi }_1^{(1)}(p_i,q_i){\varphi }_1^{(1)}(p_j,q_j)$$

$$V_{2,q}=\sum _{j\in D}{q}_j(1-q_j)({\varphi }_2^{(1)}(p_j,q_j){)}^2-2\sum _{(i,j)\in D^2,i\ne j}{q}_i{q}_j{\varphi }_2^{(1)}(p_i,q_i){\varphi }_2^{(1)}(p_j,q_j)$$

$$\sqrt{n}(J({\hat{\text{p}}}_n,\text{q})-J(\text{p},\text{q}))\stackrel{\mathcal{D}}{⇝}\mathcal{N}\left(0,V_{1,p}\right),$$

Full text

1. Introduction

1.1. Motivations

In this paper, we study the convergence of $\phi-$divergence measure estimator for empirical discrete probability distributions supported on a finite set.

Let throughout the following $\mathcal{X}=\{c_{1},c_{2},\cdots,c_{r}\}\,(r\geq 2)$ be a finite countable space. The probability distributions on $\mathcal{X}$ are finite dimensional vectors p in

[TABLE]

A divergence measure on $\mathcal{P}(\mathcal{X})$ is a function

[TABLE]

such that $\mathcal{D}(\textbf{p},\textbf{p})=0$ for any p such that $(\textbf{p},\textbf{p})$ in the domain of application of $\mathcal{D}$.

The function $\mathcal{D}$ is not necessarily a mapping. And if it is, it is not always symmetrical and it does neither have to be a metric. In lack of symmetry, the following more general notation is more appropriate :

[TABLE]

where $\mathcal{P}_{1}(\mathcal{X})$ and $\mathcal{P}_{2}(\mathcal{X})$ are two families of probability distributions on $\mathcal{X}$, not necessarily the same. To better explain our concern, let us introduce some of the most celebrated divergence measures.

Let $(\textbf{p},\textbf{q})\in\mathcal{P}(\mathcal{X})\times\mathcal{P}(\mathcal{X})$ with $\mathcal{X}=\{c_{1},c_{2},\cdots,c_{r}\}$, and let $X$ and $Y$ two randoms variables such that

[TABLE]

and set $\textbf{p}=(p_{1},\cdots,p_{r})^{t}$ and $\textbf{q}=(q_{1},\cdots,q_{r})^{t}$.

The four most popular divergence are :

(1) The $L_{2}^{2}$-divergence measure :

[TABLE]

(2) The family of Renyi’s divergence measures indexed by $\alpha\neq 1$, $\alpha>0$, known under the name of Renyi-$\alpha$ :

[TABLE]

(3) The family of Tsallis divergence measures indexed by $\alpha\neq 1$, $\alpha>0$, also known under the name of Tsallis-$\alpha$ :

[TABLE]

(4) The Kulback-Leibler divergence measure

[TABLE]

The latter, the Kullback-Leibler divergence measure, may be interpreted as a limit case of both the Renyi’s family and the Tsallis’ one by letting $\alpha\rightarrow 1$. As well, for $\alpha$ near 1, the Tsallis family may be seen as derived from $\mathcal{D}_{R,\alpha}(\textbf{p},\textbf{q})$ based on the first order expansion of the logarithm function in the neighborhood of the unity. Here for ease of notation we refer the notation log as the natural logarithm

From this small sample of divergence measures, we may give the following remarks :

For both the Renyi and the Tsallis families, we may have computation problems. So without loss of generality, suppose

[TABLE]

If Assumption (1.7) holds, we do not have to worry about summation problems, especially for Tsallis, Renyi and Kulback-Leibler measures, in the computations arising in estimation theories. This explains why Assumption (1.7) is systematically used in a great number of works in that topic, for example, in Singh and Poczos (2014), Krishnamurthy et al. (2014), Hall (1987), and recently in Bâ et al. (2018) to cite a few.

It is clear from the very form of these divergence measures that we do not have symmetry, unless for the special case where $\alpha=1/2$. So we define the following symetric version of divergence measures

[TABLE]

provided that $\mathcal{D}(\textbf{p},\textbf{q})$ and $\mathcal{D}(\textbf{q},\textbf{p})$ are finite.

Both families are build on the following summation

[TABLE]

Although we are focusing on the aforementioned divergence measures in this paper, it is worth mentioning that there exist quite a few number of them. Let us cite for example the ones named after : Ali-Silvey or $f$-divergence Topsoe (2000), Cauchy-Schwarz, Jeffrey divergence (see Evren (2012)), Chernoff (See Evren (2012)) , Jensen-Shannon (See Evren (2012)). According to Cichocki and Amari (2010), there is more than a dozen of different divergence measures in the literature.

Before coming back to our divergence measures estimation of interest, we want to highlight some important applications of them. Indeed, divergence has proven to be useful in applications. Let us cite a few of them :

(a) They heavily intervene in Information Theory and recently in Machine Learning.

(b) They have been used as similarity measures in image registration or multimedia classification (see Moreno et al. (2004)).

(c) They are also used as loss functions in evaluating and optimizing the performance of density estimation methods (see Hall (1987)).

(d) Divergence estimates can also be used to determine sample sizes required to achieve given performance levels in hypothesis testing.

(e) There has been a growing interest in applying divergence to various fields of science and engineering for the purpose of estimation, classification, etc. (See Bhattacharya (1967), Liu and Shum (2003)).

(f) Divergence also plays a central role in the frame of large deviations results including the asymptotic rate of decrease of error probability in binary hypothesis testing problems.

(g) The estimation of divergence between the samples drawn from unknown distributions gauges the distance between those distributions. Divergence estimates can then be used in clustering and in particular for deciding whether the samples come from the same distribution by comparing the estimate to a threshold.

(h) Divergence gauges how differently two random variables are distributed and it provides a useful measure of discrepancy between distributions. In the frame of information theory, the key role of divergence is well known.

The reader may find more applications and descriptions in the following papers : Kullback and Leibler (1951),Fukunaga and Hayes (1989), Cardoso (1997), Ojala et al. (1996), Hastie and Tibshirani (1998), Moreno et al. (2004),MacKay (2003).

In the next subsection, we describe the frame in which we place the estimation problems we deal in this paper.

1.2. Statistical Estimations

The divergence measures may be applied to two statistical problems among others.

(A) First, it may be used as a fitting problem as described here. Let $X_{1},X_{2},\cdots$ a sample of replications of $X$ with an unknown probability distribution p and we want to test the hypothesis that p is equal to a known and fixed probability $\textbf{p}_{0}.$ Theoretically, we can answer this question by estimating a divergence measure $\mathcal{D}(\textbf{p},\textbf{p}_{0})$ by a plug-in estimator $\mathcal{D}(\widehat{\textbf{p}}_{n},\textbf{p})$ where, for each $n\geq 1$, p is replaced by an estimator $\widehat{\textbf{p}}_{n}$ of the probability law, which is based on sample $X_{1}$, $X_{2}$, …, $X_{n}$, to be precised.

From there establishing an asymptotic theory of $\Delta_{n}=\mathcal{D}(\widehat{\textbf{p}}_{n},\textbf{p}_{0})-\mathcal{D}(\textbf{p},\textbf{p}_{0})$ is thought to be necessary to conclude.

(B) Next, it may be used as tool of comparing for two distributions. We may have two samples and wonder whether they come from the same probability distribution. Here, we also may two different cases.

(B1) In the first, we have two independent samples $X_{1},X_{2},....$ and $Y_{1},Y_{2},....$ respectively from a random variable $X$ and $Y$ according the probability distributions p and q. Here the estimated divergence $\mathcal{D}(\widehat{\textbf{p}}_{n},\widehat{\textbf{q}}_{m})$, where $n$ and $m$ are the sizes of the available samples, is the natural estimator of $\mathcal{D}(\textbf{p},\textbf{q})$ on which depends the statistical test of the hypothesis : $\textbf{p}=\textbf{q}$.

(B2) But the data may also be paired $(X,Y)$, $(X_{1},Y_{2}),(X_{2},Y_{2}),...,$ that is $X_{i}$ and $Y_{i}$ are measurements of the same case $i=1,2,...$ In such a situation, testing the equality of the margins $\textbf{p}_{X}=\textbf{p}_{Y}$ should be based on an estimator $\widehat{\textbf{p}}_{X,Y}^{(n)}$ of the joint probability law of the couple $(X,Y)$ based of the paired observations $(X_{i},Y_{i})$, $i=1,2,\ldots,n$.

We did not encounter the approach (B2) in the literature. In the (B1) approach, almost all the papers used the same sample size, at the exception of Poczos and Jeff (2011), for the double-size estimation problem. In our view, the study case should rely on the available data so that using the same sample size may lead to a loss of information. To apply their method, one should take the minimum of the two sizes and then loose information. We suggest to come back to a general case and then study the asymptotic theory of $\mathcal{D}(\widehat{\textbf{p}}_{n},\widehat{\textbf{q}}_{m})$ based on samples $X_{1},X_{2},..,X_{n}.$ and $Y_{1},Y_{2},...,Y_{m}$. In this paper, we will systematically use arbitrary samples sizes.

1.3. Previous work

In the context of the situation (B1), there are several papers dealing with the estimation of the divergence measures. As we are concerned in this paper by the weak laws of the estimators, our review on that problematic did not return significant things. Instead, the literature presented us many kinds of results on almost-sure efficiency of the estimation, with rates of convergences and laws of the iterated logarithm, $L^{p}$ ($p=1,2$) convergences, etc. To be precise, Dhakher et al. (2016) used recent techniques based on functional empirical process to provide a series of interesting rates of convergence of the estimators in the case of one-sided approach for the class de Renyi, Tsallis, Kullback-Leibler to cite a few. Unfortunately, the authors did not address the problem of integrability, taking that the divergence measures are finite. Although the results should be correct under the boundedness assumption (1.7) (BD) we described earlier, a new formulation in that frame would be welcome.

In the context of the situation (B1), we may cite first the works of Krishnamurthy et al. (2014) and Singh and Poczos (2014). They both used divergence measures based on probability density functions and concentrated of Renyi-$\alpha$, Tsallis-$\alpha$ and Kullback-Leibler.

Specifically, Krishnamurthy et al. (2014) defined Reyni and Tsallis estimators by correcting the plug-in estimator and established that, as long as $\mathcal{D}_{R,\alpha}(\textbf{p},\textbf{q})\geq c$ and $\mathcal{D}_{T,\alpha}(\textbf{p},\textbf{q})\geq c$, for some constant $c>0$, then

[TABLE]

Poczos and Jeff (2011) used a $k-$nearest-neighbor approach to prove that if $|\alpha-1|<k$, ($\alpha\neq 1)$ then

[TABLE]

There has been recent interest in deriving convergence rates for divergence estimators Moon and Hero (2014)-Krishnamurthy et al. (2014). The rates are typically derived in terms of smoothness $s$ of the densities :

The estimator of Liu et al. (2012) converges at rate $n^{-\frac{s}{s+d}}$, achieving the parametric rate when $s>d$.

Similarly, Sricharan et al. (2012) show that when $s>d$ a $k$-nearest-neighbor style estimator achieves rate $n^{-2/d}$ (in absolute error) ignoring logarithmic factors. In a follow up work, the authors improve this result to $O(n^{-1/2})$ using an ensemble of weak estimators, but they require $s>d$ orders of smoothness.

Singh and Poczos (2014) provided an estimator for Rényi$-\alpha$ divergences as well as general density functionals that uses a mirror image kernel density estimator. They obtained exponential inequalities for the deviation of the estimators from the true value.

Kallberg and Seleznjev (2012) studied an $\varepsilon-$nearest neighbor estimator for the $L_{2}-$divergence that enjoys the same rate of convergence as the projection-based estimator of Krishnamurthy et al. (2014).

1.4. Main contributions

Our main contribution may be summurized as follows, for data sampled from one or two unknown random variables, we derive almost sure convergency and central limit theorems for empirical $\phi-$ divergences. We will focus on divergence measures between discrete probability distribution. As well, our results applied to the approaches (A) and (B1) defined above. As a consequence, we estimate divergence measures by their plug-in counterparts, meaning that we replace the probability mass function (p.m.f.) in the expression of the divergence measure by a nonparametric estimator of the p.m.f.’s.

We also wish to get first general laws for an arbitrary functional of the form

[TABLE]

where $\phi:(0,1)^{2}\rightarrow\mathbb{R}$ is a twice continously differentiable function. The results on the functional $J(\textbf{p},\textbf{q})$, which is also known under the name of $\phi$-divergence, will lead to those on the particular cases of the Tsallis and Kullback-Leibler measures.

1.5. Overview of the paper

The rest of the paper is organized as follows. In Subsection 2.1, we define estimators of the p.m.f. $p_{j}$ and $q_{j}$ based on i.i.d. samples according respectively to p and q. In Section 3, we will give our foul results for functional $J(\textbf{p},\textbf{q})$ both one sided and two-sided approaches. In Section 4, we will particularize the results for specific measures we already described. Section 5 provides the proofs and in Section 6 we present some simulations confirming our results. Finally in Section 7, we conclude.

2. Empirical $\phi-$ divergence

2.1. Notations and main results

Before we state the main results we need a few definitions. Let $X$ and $Y$ two randoms variables defined on the probability distributions $(\mathcal{X},\mathcal{A},\mathbb{P})$ with $\mathcal{X}=\{c_{1},c_{2},\cdots,c_{r}\}$ and $\mathbf{p}=(p_{j})_{1\leq j\leq r}$ and $\mathbf{q}=(q_{j})_{1\leq j\leq r}$ two discrete probability distributions on $\mathcal{X}$ such that, for any $j\in D=\{1,2,\cdots,r\}$

[TABLE]

We suppose that (1.7) is satisfied that is $\forall\,j\in D,\ \ p_{j}>0\ \ \text{and}\ \ q_{j}>0.$

Define the empirical probability distribution generated by i.i.d. random variables $X_{1},\cdots,X_{n}$ from the probability distribution p as

[TABLE]

where $1_{c_{j}}(X_{i})=\begin{cases}1\ \ \text{if}\ \ X_{i}=c_{j}\\ 0\ \ \text{otherwise}\end{cases}$ for any $j\in\{1,\cdots,r\}$.

$\widehat{\textbf{q}}_{m}$ is defined in the same way by $Y_{1},\cdots,Y_{m}\stackrel{{\scriptstyle i.i.d.}}{{\sim}}\textbf{q}$ that is

[TABLE]

2.2. $\phi$-divergence measure

Definition 1**.**

The $\phi$-divergence between the two probability distributions p and q is given by

[TABLE]

where $\phi:[0,1]^{2}\rightarrow\mathbb{R}$ is a measurable function having continuous second order partial derivatives.

The results on the functional $J(\textbf{p},\textbf{q})$ will lead to those on the particular cases of the Renyi, Tsallis, and Kullback-Leibler measures.

Based on (2.1) and (2.2), we will use the following empirical $\phi$-divergences.

[TABLE]

Set

[TABLE]

Denote

[TABLE]

and

[TABLE]

Set

[TABLE]

3. Statements of the main results

3.1. Main results

Here are our main results. The first concerns the almost sure efficiency of the estimators.

Theorem 1**.**

Let p and q two probability distributions and $\widehat{\textbf{p}}_{n}$ and $\widehat{\textbf{q}}_{m}$ be generated by i.i.d. samples $X_{1},\cdots,X_{n}$ and $Y_{1},\cdots,Y_{m}$ according respectively to p and q and given by (2.1) and (2.2), $(BD)$ (1.7) be satisfied. Then the following asymptotic results hold

• (a)

One sample

[TABLE]

• (b)

Two samples :

[TABLE]

where $a_{n}$, $b_{m}$ and $c_{n,m}$ are as in (2.4) and $A_{1,p}$ and $A_{2,q}$ as in (2.5).

The second concerns the asymptotic normality of the estimators.

Let

[TABLE]

and

[TABLE]

Theorem 2**.**

Under the same assumptions as in Theorem 1, the following central limit theorems hold.

• (a)

One sample : as $n\rightarrow+\infty$,

[TABLE]

[TABLE]

• (b)

Two samples : for $(n,m)\rightarrow(+\infty,+\infty)$ and $nm/(n+m)\rightarrow\gamma\in(0,1)$,

[TABLE]

3.2. Direct extensions

Quite a few number of divergence measures are not symmetrical. Among these non-symmetrical measures are some of the most interesting ones. For such measures, estimators of the form $J(\widehat{\textbf{p}}_{n},\textbf{q})$, $J(\textbf{p},\widehat{\textbf{q}}_{m})$ and $J(\widehat{\textbf{p}}_{n},\widehat{\textbf{q}}_{m})$ are not equal to $J(\textbf{q},\widehat{\textbf{p}}_{n})$, $J(\widehat{\textbf{q}}_{m},\textbf{p})$ and $J(\widehat{\textbf{q}}_{m},\widehat{\textbf{p}}_{n})$ respectively.

In one-sided tests, we have to decide whether the hypothesis $\textbf{p}=\textbf{q}$, for q known and fixed, is true based on data from p. In such a case, we may use the statistics one of the statistics $(J(\widehat{\textbf{p}}_{n},\textbf{q})$ and $J(\textbf{q},\widehat{\textbf{p}}_{n})$ to perform the tests. We may have information that allows us to prefer one of them. If not, it is better to use both of them, upon the finiteness of both $J(\textbf{p},\textbf{q})$ and $J(\textbf{q},\textbf{p})$, in a symmetrized form as

[TABLE]

The same situation applies when we face double-side tests, i.e., testing $\textbf{p}=\textbf{q}$ from data generated by p et q.

Asymptotic a.e. efficiency.

Theorem 3**.**

Under the same assumptions as in Theorem 1, the following hold

• (a)

One sample :

[TABLE]

[TABLE]

• (b)

Two samples :

[TABLE]

Asymptotic Normality.

Denote

[TABLE]

and finally

[TABLE]

We have

Theorem 4**.**

Under the same assumptions as in Theorem 1, the following hold.

• (a)

One sample : as $n\rightarrow+\infty$,

[TABLE]

[TABLE]

• (b)

Two samples : for $(n,m)\rightarrow(+\infty,+\infty)$ and $nm/(n+m)\rightarrow\gamma\in(0,1)$,

[TABLE]

Remark : The proof of these extensions will not be given here, since they are straight consequences of the main results. As well, such considerations will not be made again for particular measures for the same reason.

4. Particular Cases

4.1. Renyi and Tsallis families

These two families are expressed through the summation

[TABLE]

which is of the form of the $\phi-$divergence measure with

[TABLE]

A-(a)- The asymptotic behavior of the Tsallis divergence measure.

Denote

[TABLE]

We have

Corollary 1**.**

Under the same assumptions as in Theorem 1, and for any $\alpha>0,\ \alpha\neq 1$, the following hold

• (a)

One sample :

[TABLE]

• (b)

Two samples :

[TABLE]

Denote

[TABLE]

We have

Corollary 2**.**

Under the same assumptions as in Theorem 1, and for any $\alpha>0,\ \alpha\neq 1$, the following hold

• (a)

One sample : as $n\rightarrow+\infty$

[TABLE]

[TABLE]

• (b)

Two samples : for $(n,m)\rightarrow(+\infty,+\infty)$ and $nm/(n+m)\rightarrow\gamma\in(0,1)$,

[TABLE]

As to the symmetrized form

[TABLE]

we need the supplementaries notations:

[TABLE]

[TABLE]

We have

Corollary 3**.**

Under the same assumptions as in Theorem 1, and for any $\alpha>0,\ \alpha\neq 1$, the following hold

• (a)

One sample :

[TABLE]

[TABLE]

• (b)

Two samples :

[TABLE]

Denote

[TABLE]

We also have

Corollary 4**.**

Under the same assumptions as in Theorem 1, and for any $\alpha>0,\ \alpha\neq 1$, the following hold

• (a)

One sample : as $n\rightarrow+\infty,$

[TABLE]

[TABLE]

• (b)

Two samples : for $(n,m)\rightarrow(+\infty,+\infty)$ and $nm/(n+m)\rightarrow\gamma\in(0,1)$,

[TABLE]

A-(b)- The asymptotic behavior of the Renyi-$\alpha$ divergence measure.

The treatment of the asymptotic behavior of the Renyi-$\alpha$, $\alpha>0$, $\alpha\neq 1$ is obtained from Part (A)-(a) by expansions and by the application of the delta method.

We first remark that

[TABLE]

Corollary 5**.**

Under the same assumptions as in Theorem 1, and for any $\alpha>0,\ \alpha\neq 1$, the following hold

• (a)

One sample :

[TABLE]

[TABLE]

• (b)

Two samples :

[TABLE]

Denote

[TABLE]

We have

Corollary 6**.**

Under the same assumptions as in Theorem 1, and for any $\alpha>0,\ \alpha\neq 1$, the following hold

• (a)

One sample : as $n\rightarrow+\infty$

[TABLE]

[TABLE]

• (b)

Two samples : for $(n,m)\rightarrow(+\infty,+\infty)$ and $nm/(n+m)\rightarrow\gamma\in(0,1)$,

[TABLE]

As to the symetrized form

[TABLE]

we need the supplementary notations :

[TABLE]

Corollary 7**.**

Under the same assumptions as in Theorem 1, and for any $\alpha>0,\ \alpha\neq 1$, the following hold.

• (a)

One sample :

[TABLE]

• (b)

Two samples :

[TABLE]

Denote

[TABLE]

We also have

Corollary 8**.**

Under the same assumptions as in Theorem 1, and for any $\alpha>0,\ \alpha\neq 1$, the following hold.

• (a)

One sample : as $n\rightarrow+\infty$,

[TABLE]

• (b)

Two samples : as $(n,m)\rightarrow(+\infty,+\infty)$ and $nm/(n+m)\rightarrow\gamma\in(0,1)$,

[TABLE]

**B - Kulback-Leibler Measure

**

Here we have

[TABLE]

where

[TABLE]

So we have first :

Corollary 9**.**

Under the same assumptions as in Theorem 1, the following hold.

• (a)

One sample :

[TABLE]

• (b)

Two samples :

[TABLE]

Denote

[TABLE]

We have

Corollary 10**.**

Under the same assumptions as in Theorem 1, the following hold.

• (a)

One sample : as $n\rightarrow+\infty$

[TABLE]

• (b)

Two samples : as $(n,m)\rightarrow(+\infty,+\infty)$ and $nm/(n+m)\rightarrow\gamma\in(0,1)$,

[TABLE]

As to the symmetrized form

[TABLE]

we need the supplementary notations :

[TABLE]

We have

Corollary 11**.**

Under the same assumptions as in Theorem 1, the following hold

• (a)

One sample :

[TABLE]

• (b)

Two samples :

[TABLE]

Denote

[TABLE]

and finally

[TABLE]

We also have

Corollary 12**.**

Under the same assumptions as in Theorem 1, the following hold

• (a)

One sample : as $n\rightarrow+\infty$

[TABLE]

• (b)

Two samples : for $(n,m)\rightarrow(+\infty,+\infty)$ and $nm/(n+m)\rightarrow\gamma\in(0,1)$,

[TABLE]

5. Proofs

In the proofs, we will systematically use the mean values theorem. In the multivariate handling, we prefer to use the Taylor-Lagrange-Cauchy as stated in Valiron (1966), page 230.

For sake of simplicity, we introduce the two following notations :

[TABLE]

therefore

[TABLE]

For any $j\in D$, set

[TABLE]

Before we start the proofs we recall that, since for a fixed $j\in D,$ $n\widehat{p}_{n}^{c_{j}}$ has a binomial distribution with parameters $n$ and success probability $p_{j}$, we have

[TABLE]

Furthermore, by the strong law of large numbers, we know that

[TABLE]

for a fixed $j\in D.$

And finally, by the asymptotic Gaussian limit of the multinomial law (see for example Lo et al. (2016), Chapter 1, Section 4), we have

[TABLE]

where $Z(\textbf{p})=(Z_{p_{j}},j\in D)^{t}$  and $Z(\textbf{q})=(Z_{q_{j}},j\in D)^{t}$ are two centered Gaussian random vectors of dimension $\#(D)$ which are independent and have the following elements :

[TABLE]

For a fixed $j\in D$, we have also

[TABLE]

Now we can start by showing Theorem 1.

For a fixed $j\in D$, we have

[TABLE]

by applying the mean value theorem to the function $(.)\mapsto\phi((.),q_{j})$ and where $\theta_{1,j}$ is some number lying between [math] and $1$. In the sequel, any $\theta_{i,j},\,i=1,2,\cdots$ satisfies $\left|\theta_{i,j}\right|<1$.

By applying again the mean values theorem to the function $(.)\mapsto\phi_{1}^{(1)}((.),q_{j})$, we have

[TABLE]

We can write (5) as

[TABLE]

Now we have

[TABLE]

hence

[TABLE]

Therefore

[TABLE]

We know that $A_{1,p}<\infty$ and

[TABLE]

This proves (3.1).

Formula (3.2) is obtained in a similar way. We only need to adapt the result concerning the first coordinate to the second.

The proof of (3.3) comes by splitting $\sum_{j\in D}\left(\phi(\widehat{p}_{n}^{c_{j}},\widehat{q}_{m}^{c_{j}})-\phi(p_{j},q_{j})\right)$, into the following two terms

[TABLE]

We already know how the handle $I_{n,2}$. As to $I_{n,1}$, we may still use the Taylor-Lagrange-Cauchy formula since we have, for a fixed $j\in D$,

[TABLE]

By the Taylor-Lagrange-Cauchy (see Valiron (1966), page 230), we have

[TABLE]

From there, the combination of these remarks direct to the result.$\blacksquare$

Let us prove (3.4). By going back to (5.6), we have

[TABLE]

where

[TABLE]

Now using Formula (5.1) above, we get,

[TABLE]

which follows a centered normal law of variance $V_{1,p}$ :

[TABLE]

since

[TABLE]

Let show that $\sqrt{n}R_{1,n}=0_{\mathbb{P}}(1)$. We have

[TABLE]

Let show that

[TABLE]

By the Bienaymé-Tchebychev inequality, we have, for any $\epsilon>0$ and for $j\in D$,

[TABLE]

which implies that $\sqrt{n}a_{n}^{2}$ converges in probability to [math] as $n\rightarrow+\infty$.

Finally from (5.7) we have $\sqrt{n}R_{1,n}\stackrel{{\scriptstyle\mathbb{P}}}{{\rightarrow}}0\text{ as }n\rightarrow+\infty$ which implies

[TABLE]

This ends the proof of (3.4).

The result (3.5) is obtained by a symmetry argument by swapping the role of p and $\textbf{q}.$

Now, it remains to prove Formula (3.6) of the theorem. Let us use bi-variate Taylor-Lagrange-Cauchy formula to get,

[TABLE]

where

[TABLE]

Thus we get

[TABLE]

where

[TABLE]

[TABLE]

and $R_{n,m}$ is given by

[TABLE]

First, we have that $N_{n}(\textbf{p})$ and $N_{m}(\textbf{p})$ are independents and hence

[TABLE]

Therefore

[TABLE]

Thus

[TABLE]

Next, we have

[TABLE]

That leads to

[TABLE]

since $m/(mV_{1,p}+nV_{2,q})$ and $m/(nV_{1,p}+nV_{2,q})$ are bounded, and then

[TABLE]

It remains to prove that $\left|\sqrt{\frac{nm}{mV_{1,p}+nV_{2,q}}}R_{n,m}\right|=o_{\mathbb{P}}(1).$ But we have by the continuity assumptions on $\phi$ and on its partial derivatives and by the uniform converges of $\Delta_{p_{n}}^{c_{j}}$ and $\Delta_{q_{m}}^{c_{j}}$ to zero, that

[TABLE]

As previously, we have $\sqrt{n}a_{n}^{2}=o_{\mathbb{P}}(1)$, $\sqrt{m}b_{m}^{2}=o_{\mathbb{P}}(1)$ and $\sqrt{n}a_{m}b_{m}=o_{\mathbb{P}}(1)$.

From there, the conclusion is immediate.$\blacksquare$

Proofs of Theorem 3 and Theorem 4 are obtained by writing

[TABLE]

and using Theorems 1 and 2.

6. Simulations

To assess the performance of ours estimators, we present a simulation study on a finite sample. In our simulation study, we consider tree outcomes for the randoms variables $X$ and $Y$, $c_{1},c_{2},c_{3}$ with respectives *p.m.f.*probabilities $p_{1},\ p_{2},\,p_{3}$ and $q_{1},\ q_{2},\ q_{3}$.

Our aim is to compare the performance of the divergences measures estimators as well as their symetrized forms with one or two samples when sample sizes increase.

Suppose that $p_{1}=0.4,\ p_{2}=0.25,\,p_{3}=0.35$ and $q_{1}=0.27,\ q_{2}=0.32,\ q_{3}=0.41$.

With these above values of $p_{j}$ and $q_{j}$, $j=1,2,3$, the true and the symetrized form of our interest divergence measures become

[TABLE]

In each figure, left panels represent the plots of divergence measure estimator, built from sample sizes of $n=100,200,\cdots,30000$, and the true divergence measure (represented by horizontal black line). The middle panels show the histograms of the data and where the red line represents the plots of the theoretical normal distribution calculated from the same mean and the same standard deviation of the data. The right panels concern the Q-Q plot of the data which display the observed values against normally distributed data (represented by the red line). We see that the underlying distribution of the data is normal since the points fall along a straight line.

As seen in each plots in figures 1, 2, 3, 4, 5, 6, 7, and 8 our method performs well in showing the consistency of the divergence measures estimators and the asymptotic normality through their quantile-normal graphs.

7. Conclusion

This paper joins a growing body of literature on estimating divergence measures in the discrete case and on finite sets. We adopted the plug-in method and we derived almost sure rates of convergence and asymptotic normality of the most common divergence measures in one sample, two samples as well as symetrical form of divergence measures, all this, by means of the functional $\phi-$divergence measure.