A Short Note on the Jensen-Shannon Divergence between Simple Mixture Distributions

TL;DR

This paper investigates the properties of the symmetric Jensen-Shannon divergence between two discrete mixture distributions, providing both theoretical analysis and experimental insights into how divergence varies with mixture proportions and distribution differences.

Contribution

It offers new theoretical and experimental insights into the behavior of Jensen-Shannon divergence for mixture distributions with varying components and supports.

Findings

Divergence behavior changes with mixture proportions.

Insights into divergence when supports do not coincide.

Theoretical bounds and experimental validation.

Abstract

This short note presents results about the symmetric Jensen-Shannon divergence between two discrete mixture distributions $p_1$ and $p_2$. Specifically, for $i=1,2$, $p_i$ is the mixture of a common distribution $q$ and a distribution $\tilde{p}_i$ with mixture proportion $\lambda_i$. In general, $\tilde{p}_1\neq \tilde{p}_2$ and $\lambda_1\neq\lambda_2$. We provide experimental and theoretical insight to the behavior of the symmetric Jensen-Shannon divergence between $p_1$ and $p_2$ as the mixture proportions or the divergence between $\tilde{p}_1$ and $\tilde{p}_2$ change. We also provide insight into scenarios where the supports of the distributions $\tilde{p}_1$, $\tilde{p}_2$, and $q$ do not coincide.

Figures

Tables

Table 1. TABLE I : Simulation Setting: 𝒳 = { 1 , 2 , 3 , 4 , 5 , 6 } 𝒳 1 2 3 4 5 6 \mathcal{X}=\{1,2,3,4,5,6\}

$x$	1	2	3	4	5	6
${\tilde{p}}_1\left(x\right)$	1	0	0	0	0	0
${\tilde{p}}_2\left(x\right)$	$1-\epsilon$	$\epsilon$	0	0	0	0
$q\left(x\right)$	0.5	0.4	0.025	0.025	0.025	0.025

Equations

$$H\left(r\right):=-\sum _{x\in \mathrm{supp}(r)}r(x)\log r(x)$$

$${\mathbb{D}}_{KL}(r_1\Vert r_2):=\sum _{x\in \mathrm{supp}(r_1)}{r}_1(x)\log \frac{r_1(x)}{r_2(x)}$$

$${\mathbb{D}}_{JS,\pi }(r_1,r_2)$$

$$=H\left(r_M\right)-\pi H\left(r_1\right)-(1-\pi )H\left(r_2\right)$$

$$r_M:=\pi r_1+(1-\pi )r_2.$$

$$\frac{{\left(h_2(\pi )-{\mathbb{D}}_{JS,\pi }(r_1,r_2)\right)}^2}{4}\le P_e\le \frac{h_2(\pi )-{\mathbb{D}}_{JS,\pi }(r_1,r_2)}{2}$$

$$p_1:={\lambda }_1{\tilde{p}}_1+(1-{\lambda }_1)q\text{and}{p}_2:={\lambda }_2{\tilde{p}}_2+(1-{\lambda }_2)q$$

$$\begin{gathered} {\mathbb{D}}_{JS}(p_1,p_2)={\mathbb{D}}_{JS}(({\lambda }_1,1-{\lambda }_1),({\lambda }_2,1-{\lambda }_2)) \\ +\frac{{\lambda }_1+{\lambda }_2}{2}{\mathbb{D}}_{JS,\frac{{\lambda }_1}{{\lambda }_1+{\lambda }_2}}({\tilde{p}}_1,{\tilde{p}}_2). \end{gathered}$$

$$\frac{\mathrm{d}}{\mathrm{d}\lambda }H\left(r(\cdot ,\lambda )\right)=-\sum _{x\in \mathrm{supp}(r)}\frac{\mathrm{d}}{\mathrm{d}\lambda }r(x,\lambda )\left[1+\log r(x,\lambda )\right].$$

$$p_M:=\frac{1}{2}{p}_1+\frac{1}{2}{p}_2=\frac{{\lambda }_1}{2}{\tilde{p}}_1+\frac{{\lambda }_2}{2}{\tilde{p}}_2+\frac{2-{\lambda }_1-{\lambda }_2}{2}q.$$

$$-\frac{\mathrm{d}}{\mathrm{d}\lambda }{\mathbb{D}}_{JS}(p_1,p_2)$$

$$=\sum _{x\in \mathcal{X}}\left[\frac{{\tilde{p}}_1(x)}{2}+\frac{\alpha {\tilde{p}}_2(x)}{2}-\frac{1+\alpha }{2}q(x)\right]\left(1+\log p_M(x)\right)$$

$$-\frac{1}{2}\sum _{x\in \mathrm{supp}(p_1)}\left[{\tilde{p}}_1(x)-q(x)\right]\left(1+\log p_1(x)\right)$$

$$-\frac{1}{2}\sum _{x\in \mathrm{supp}(p_2)}\left[\alpha {\tilde{p}}_2(x)-\alpha q(x)\right]\left(1+\log p_2(x)\right)$$

$$=\sum _{x\in \mathcal{X}}\left[\frac{{\tilde{p}}_1(x)}{2}+\frac{\alpha {\tilde{p}}_2(x)}{2}-\frac{1+\alpha }{2}q(x)\right]\log p_M(x)$$

$$-\frac{1}{2}\left[{\tilde{p}}_1(x)-q(x)\right]\log p_1(x)$$

$$-\frac{1}{2}\left[\alpha {\tilde{p}}_2(x)-\alpha q(x)\right]\log p_2(x)$$

$$\left[\alpha {\tilde{p}}_2(x)-\alpha q(x)\right]\log p_2(x)$$

$$=\left[\alpha {\tilde{p}}_2(x)-\alpha q(x)\right]\log \left(\alpha \lambda {\tilde{p}}_2(x)+(1-\alpha \lambda )q(x)\right)$$

$$=\left[\alpha {\tilde{p}}_2(x)-\alpha q(x)\right]\log \left(\lambda (\alpha {\tilde{p}}_2(x)-\alpha q(x))+q(x)\right)$$

$$\begin{gathered} {\mathbb{D}}_{JS}(p_1,p_2)=H\left(\frac{2\lambda +\Delta \lambda }{2}\tilde{p}+\frac{2-2\lambda -\Delta \lambda }{2}q\right) \\ -\frac{1}{2}H\left(\lambda \tilde{p}+(1-\lambda )q\right)-\frac{1}{2}H\left((\lambda +\Delta \lambda )\tilde{p}+(1-\lambda -\Delta \lambda )q\right) \end{gathered}$$

$$\frac{\mathrm{d}}{\mathrm{d}\Delta \lambda }{\mathbb{D}}_{JS}(p_1,p_2)$$

$$=\sum _{x\in \mathcal{X}}\left[\frac{\tilde{p}(x)-q(x)}{2}\right]$$

$$\times \log \left(\frac{(\lambda +\Delta \lambda )\tilde{p}(x)+(1-\lambda -\Delta \lambda )q(x)}{(\lambda +\frac{\Delta \lambda }{2})\tilde{p}(x)+(1-\lambda -\frac{\Delta \lambda }{2})q(x)}\right)$$

$$=\sum _{x\in \mathcal{X}}\left[\frac{\tilde{p}(x)-q(x)}{2}\right]$$

$$\times \log \left(1+\frac{\Delta \lambda }{2}\frac{\tilde{p}(x)-q(x)}{(\lambda +\frac{\Delta \lambda }{2})\tilde{p}(x)+(1-\lambda -\frac{\Delta \lambda }{2})q(x)}\right)$$

$$=\sum _{x\in \mathcal{X}}\left[\frac{\tilde{p}(x)-q(x)}{2}\right]\log \left(1+\frac{\Delta \lambda }{2}\frac{\tilde{p}(x)-q(x)}{p_M(x)}\right).$$

$$\frac{\mathrm{d}}{\mathrm{d}\Delta \lambda }{\mathbb{D}}_{JS}(p_1,p_2)$$

$$\ge \sum _{x:\tilde{p}(x)\ge q(x)}{\left[\frac{\tilde{p}(x)-q(x)}{2}\right]}^2\frac{\frac{\Delta \lambda }{p_M(x)}}{1+\frac{\Delta \lambda }{2}\frac{\tilde{p}(x)-q(x)}{p_M(x)}}$$

$$+\sum _{x:\tilde{p}(x)<q(x)}{\left[\frac{\tilde{p}(x)-q(x)}{2}\right]}^2\frac{\Delta \lambda }{p_M(x)}$$

$$\ge 0$$

$${\mathbb{D}}_{KL}(p_1\Vert p_M)$$

$$=\sum _{x\in \mathrm{supp}({\tilde{p}}_1)}{\lambda }_1{\tilde{p}}_1(x)\log \frac{{\lambda }_1{\tilde{p}}_1(x)}{\frac{{\lambda }_1}{2}{\tilde{p}}_1(x)+\frac{{\lambda }_2}{2}{\tilde{p}}_2(x)}$$

$$+\sum _{x\in \mathrm{supp}(q)}(1-{\lambda }_1)q(x)\log \frac{(1-{\lambda }_1)q(x)}{\frac{2-{\lambda }_1-{\lambda }_2}{2}q(x)}$$

$$={\lambda }_1\sum _{x\in \mathrm{supp}({\tilde{p}}_1)}{\tilde{p}}_1(x)\log \frac{{\tilde{p}}_1(x)}{\frac{{\lambda }_1}{{\lambda }_1+{\lambda }_2}{\tilde{p}}_1(x)+\frac{{\lambda }_2}{{\lambda }_1+{\lambda }_2}{\tilde{p}}_2(x)}$$

$$+(1-{\lambda }_1)\log \frac{(1-{\lambda }_1)}{\frac{2-{\lambda }_1-{\lambda }_2}{2}}+{\lambda }_1\log \frac{{\lambda }_1}{\frac{{\lambda }_1+{\lambda }_2}{2}}$$

$$={\lambda }_1{\mathbb{D}}_{KL}({\tilde{p}}_1\Vert \tilde{p})+{\mathbb{D}}_{KL}(({\lambda }_1,1-{\lambda }_1)\Vert \lambda ,1-\lambda ))$$

Full text

Motivation

Suppose there are three types of dice (red, blue, and green), each of which is characterized by a specific probability distribution over its faces. Suppose further that there are two urns (A and B), of which A contains only red and green dice, while B contains only blue and green dice. The proportion of red dice in A and of blue dice in B shall be known. Suppose finally that a player randomly chooses one of the urns, picks a die from the chosen urn at random, and rolls the die. You only observe the outcome of the die roll. In the problem of guessing, given only the observed die roll, the urn from which the die was chosen, the Jensen-Shannon (JS) divergence plays a significant role in bounds on the probability of guessing correctly. In this short note we evaluate the JS divergence as a function of the probability distribution over the faces of the three types of dice, and as a function of the proportion of dice of a given color in the respective urn.

I Notation and Assumptions

We consider probability mass functions (PMFs) on a common finite alphabet $\mathcal{X}$, i.e., all PMFs in this work are $\mathcal{X}\to[0,1]$. Let $\mathrm{supp}(\cdot)$ denote the support of a PMF; e.g., for PMF $r$, $\mathrm{supp}(r):=\{x\in\mathcal{X}{:}\ r(x)>0\}\subseteq\mathcal{X}$.

We denote the entropy of a discrete random variable (RV) with PMF $r$ and the Kullback-Leibler divergence between two PMFs $r_{1}$ and $r_{2}$ as

[TABLE]

JS divergence admits an operational characterization in binary classification. Specifically, let $\pi$ be the prior probability of class 1, and let $1-\pi$ be the prior probability of class 2. Let further $r_{i}$ be the feature distribution under class $i$. Then, JS divergence appears in upper and lower bounds on the error probability $P_{e}$ of feature-based classification, i.e., [2, Th. 4 & 5]

[TABLE]

where $h_{2}(x):=-x\log x-(1-x)\log(1-x)$.

II JS Divergence between Simple Mixture Distributions

Suppose that $\tilde{p}_{1}$, $\tilde{p}_{2}$, and $q$ are PMFs on the common finite alphabet $\mathcal{X}$ and let $\lambda_{1},\lambda_{2}\in[0,1]$. We define

[TABLE]

i.e., the two distributions are mixtures of a common and a potentially different distribution. We consider the symmetric JS divergence between these distributions, abbreviating $\mathbb{D}_{JS,\frac{1}{2}}(p_{1},p_{2})=:\mathbb{D}_{JS}(p_{1},p_{2})$. Our first observation is negative.

Observation 1**.**

$\mathbb{D}_{JS}(p_{1},p_{2})$* is neither monotonic in the mixture proportions $\lambda_{1}$ and $\lambda_{2}$, nor in the difference between mixture proportions $|\lambda_{1}-\lambda_{2}|$, nor in the divergence between $\tilde{p}_{1}$ and $\tilde{p}_{2}$.*

While obviously $\mathbb{D}_{JS}(p_{1},p_{2})=0$ if $\lambda_{1}=\lambda_{2}=0$, it is not true that increasing the mixture proportion of one distribution relative to the other increases the symmetric JS divergence. An intuitive explanation for this is that if $\tilde{p}_{1}$ and $\tilde{p}_{2}$ are similar, then the mixture proportions $\lambda_{1}$ and $\lambda_{2}$ should also be similar to minimize the symmetric JS divergence.

Consider for example the setting in Table I with $\varepsilon=0.3$. Fig. 1(a) shows the JS divergence as a function of the mixture proportions. Evaluating this plot at specific lines yields Fig. 1(d). One can see that, fixing $\lambda_{1}=0.3$, $\mathbb{D}_{JS}(p_{1},p_{2})$ achieves its minimum for $\lambda_{2}\approx 0.5$. Similarly, for a fixed $\lambda_{2}=0.7$, the JS divergence is minimized for $\lambda_{1}\approx 0.3$. Consequently, for these settings $\mathbb{D}_{JS}(p_{1},p_{2})$ is not monotonic in the mixture proportions.

Similar considerations hold for the difference between mixture proportions and the divergence between $\tilde{p}_{1}$ and $\tilde{p}_{2}$. For example, apropriately setting $\lambda_{1}$ and $\lambda_{2}$ can compensate the effect of $\tilde{p}_{1}$ and $\tilde{p}_{2}$ being unequal. This situation is depicted in Fig. 1(e) where Fig. 1(a) is evaluated at $\lambda_{2}=0.7$ and at various values of $\lambda_{1}\in[0,0.7]$. It can be seen that the optimal value of $\lambda_{1}$ in this range is not $0.7$ (for which $|\lambda_{1}-\lambda_{2}|=0$), but close to $0.3$.

In analogy, $\tilde{p}_{1}$ and $\tilde{p}_{2}$ being different can compensate the effect of a difference between $\lambda_{1}$ and $\lambda_{2}$. Figure 1(b) displays this behavior for the example in Table I. As it can be seen, for $(\lambda_{1},\lambda_{2})=(0.3,0.7)$, the symmetric JS divergence is not minimized for $\varepsilon=0$ in which case $\tilde{p}_{1}=\tilde{p}_{2}$, but for $\varepsilon\approx 0.2$.

We next present a few positive results regarding the monotonicity of $\mathbb{D}_{JS}(p_{1},p_{2})$ and the behavior of $\mathbb{D}_{JS}(p_{1},p_{2})$ in case the supports of $\tilde{p}_{i}$ and $q$ are disjoint. The proofs are deferred to Section III.

Inspecting Fig. 1(a) suggests that $\mathbb{D}_{JS}(p_{1},p_{2})$ increases as $(\lambda_{1},\lambda_{2})$ increase jointly along a line through the origin. This behavior is also displayed in Figs. 1(d) and 1(f). We establish it as a general fact in the following observation.

Observation 2**.**

If $(\lambda_{1},\lambda_{2})=(\lambda,\alpha\lambda)$ for some $\alpha>0$, then $\mathbb{D}_{JS}(p_{1},p_{2})$ increases monotonically with $\lambda$.

Intuitively, if $\tilde{p}_{1}=\tilde{p}_{2}$, then the symmetric JS divergence between $p_{1}$ and $p_{2}$ should increase if the difference between mixture proportions increases.

Observation 3**.**

If $\tilde{p}_{1}=\tilde{p}_{2}$, then $\mathbb{D}_{JS}(p_{1},p_{2})$ increases monotonically with $|\lambda_{1}-\lambda_{2}|$.

We finally evaluate the scenario where the supports of $\tilde{p}_{1}$ and $\tilde{p}_{2}$ are disjoint from the support of $q$. Suppose we draw a sample of either $p_{1}$ or $p_{2}$ and it is our task to determine from which distribution it was drawn. We assume to know the supports of $\tilde{p}_{1}$, $\tilde{p}_{2}$, and $q$. If the drawn sample is from the support of $q$, then there is now way to distinguish between $p_{1}$ and $p_{2}$; one only has the chance to distinguish $p_{1}$ from $p_{2}$ if the drawn sample is from the supports of $\tilde{p}_{1}$ and $\tilde{p}_{2}$. The following observation shows that in this case the JS divergence does not depend on the PMF $q$.

Observation 4**.**

If $\mathrm{supp}(\tilde{p}_{i})\cap\mathrm{supp}(q)=\emptyset$ for $i=1,2$, then

[TABLE]

An example for this scenario is depicted in Figs. 1(c) and 1(f). The simulation setting coincides with the one in Table I, with $\varepsilon=0.3$ and $q$ replaced by a uniform distribution on $\{3,4,5,6\}$. Obviously, since the supports of $\tilde{p}_{i}$ and $q$, $i=1,2$ are disjoint, $\mathbb{D}_{JS}(p_{1},p_{2})$ achieves its maximum at $(\lambda_{1},\lambda_{2})=(0,1)$ and $(\lambda_{1},\lambda_{2})=(1,0)$.

Now suppose $\tilde{p}_{1}=\tilde{p}_{2}$. In this case, $p_{1}$ and $p_{2}$ can only be distinguished if they mix $\tilde{p}_{1}$ and $\tilde{p}_{2}$ with different proportions (since otherwise $p_{1}=p_{2}$). This is accounted for in the first term of (3). Next, suppose that $\lambda_{1}=\lambda_{2}=\lambda$. The only chance to distinguish $p_{1}$ and $p_{2}$ results from $\tilde{p}_{1}$ and $\tilde{p}_{2}$ being different, which is accounted for in the second term of (3). Moreover, in this case $p_{1}$ and $p_{2}$ can be distinguished more easily if the common mixture proportion is large. Since finally in this case we have $\mathbb{D}_{JS}(p_{1},p_{2})=\lambda\mathbb{D}_{JS}(\tilde{p}_{1},\tilde{p}_{2})$, one can see that $\mathbb{D}_{JS}(p_{1},p_{2})$ increases linearly with $\lambda$ if $\lambda_{1}=\lambda_{2}=\lambda$ (see Fig. 1(f)). In Fig. 1(f) one can moreover see that, in the more general setting of $\lambda_{1}\neq\lambda_{2}$ and $\tilde{p}_{1}\neq\tilde{p}_{2}$, a monotonic increase is not observable. This is due to the nontrivial interplay between the two terms in (3).

III Proofs

Suppose that $r(\cdot,\lambda)$ is a PMF parameterized by $\lambda$. Simple calculus shows that

[TABLE]

Let further

[TABLE]

Proof of Observation 2.

With (1) and (4) we obtain

[TABLE]

where the last equality is obtained by continuous extension of the logarithm to obtain $0\log 0:=0$. Observe further that

[TABLE]

i.e., it is of shape $r\log(\lambda r+q)$. It can be shown that this function is convex in $r$ if $r$, $q$, and $\lambda$ are nonnegative. It thus follows that each summand in (6) is nonpositive. This completes the proof. ∎

Proof of Observation 3.

We first assume that $\lambda_{1},\lambda_{2}\in(0,1)$. We treat the remaining cases separately. Let w.l.o.g. $\lambda:=\min\{\lambda_{1},\lambda_{2}\}$ and $\Delta\lambda:=|\lambda_{2}-\lambda_{1}|$. We obtain that

[TABLE]

and thus

[TABLE]

Note that the argument of the logarithm in (10) is equivalent to the argument of the logarithm in (8), which is positive for $x\in\mathcal{X}$ if $\lambda_{1},\lambda_{2}\in(0,1)$. We can therefore break the sum into to parts, one in which $\tilde{p}\geq q$ and one in which $\tilde{p}<q$, and bound the logarithm by $x-1\geq\log x\geq\frac{x-1}{x}$. Thus we obtain

[TABLE]

where the last line follows since each summand is nonnegative.

If $(\lambda_{1},\lambda_{2})=(0,0)$, then $\mathbb{D}_{JS}(p_{1},p_{2})=0$ and (3) holds. If $(\lambda_{1},\lambda_{2})=(1,1)$, then (3) holds trivially. If $(\lambda_{1},\lambda_{2})=(1,0)$, then the supports of $p_{1}$ and $p_{2}$ are disjoint and $\mathbb{D}_{JS}(p_{1},p_{2})=\log(2)$, i.e., it achieves its maximum value. The same value is achieved by the first term in (3), while the second term can be shown to be zero in this case. This completes the proof. ∎

Proof of Observation 4.

We get

[TABLE]

where $\tilde{p}\triangleq\frac{\lambda_{1}}{\lambda_{1}+\lambda_{2}}\tilde{p}_{1}+\frac{\lambda_{2}}{\lambda_{1}+\lambda_{2}}\tilde{p}_{2}$ and where $\lambda\triangleq\frac{\lambda_{1}+\lambda_{2}}{2}$. The proof is completed by repeating the same steps for $\mathbb{D}_{KL}(p_{2}\|p_{M})$ and inserting it in (1). ∎

Acknowledgments

The author thanks Roman Kern, Know-Center GmbH and Institute for Interactive Systems and Data Science, Graz University of Technology, for fruitful discussions.