Existence and Continuity of Differential Entropy for a Class of Distributions

TL;DR

This paper identifies a class of distributions for which differential entropy is uniformly convergent under total variation distance, providing a practical way to verify entropy properties across this class.

Contribution

It introduces a new class of distributions with verifiable conditions ensuring uniform convergence of differential entropy.

Findings

Differential entropy converges uniformly over the class under total variation.

The class's conditions are easy to verify for specific distributions.

Provides theoretical foundation for entropy analysis in continuous distributions.

Abstract

In this paper, we identify a class of absolutely continuous probability distributions, and show that the differential entropy is uniformly convergent over this space under the metric of total variation distance. One of the advantages of this class is that the requirements could be readily verified for a given distribution.

Equations

$${\Vert p-q\Vert }_1={\int }_{\mathbb{R}}\left|p(x)-q(x)\right|\mathrm{d}x.$$

$$\Vert \nabla \log p(\mathbf{x}){\Vert }_2\le c_1\Vert \mathbf{x}{\Vert }_2+c_2,\forall \mathbf{x}\in {\mathbb{R}}^n.$$

$$|\mathrm{h}\left(\mathbf{X}\right)-\mathrm{h}\left(\mathbf{Y}\right)|\le \Delta ,$$

$$\mathrm{Pr}\left\{\mathbf{X}\in \mathcal{A}\right\}={\int }_{\mathcal{A}}p(\mathbf{x})\mathrm{d}\mathbf{x}.$$

$$\mathrm{h}\left(p\right)=\mathrm{h}\left(\mathbf{X}\right):=-{\int }_{{\mathbb{R}}^n}p(\mathbf{x})\log p(\mathbf{x})\mathrm{d}\mathbf{x},$$

$${\int }_{{\mathbb{R}}^n}\Vert \mathbf{x}{\Vert }_{\alpha }^{\alpha }p(\mathbf{x})\mathrm{d}\mathbf{x}<v,\mathrm{and}$$

$$\mathrm{ess}\underset{\mathbf{x}\in {\mathbb{R}}^n}{\sup }p(\mathbf{x})<m,$$

$${\varphi }_{n,\alpha ,v}(\mathbf{x})={\left[{\left(\frac{n}{\alpha v}\right)}^{\frac{1}{\alpha }}\frac{1}{2\Gamma \left(\frac{1}{\alpha }+1\right)}\right]}^n{\mathrm{e}}^{-\frac{n}{\alpha v}\Vert \mathbf{x}{\Vert }_{\alpha }^{\alpha }},$$

$${\Vert p_{\mathbf{X}}-p_{\mathbf{Y}}\Vert }_1={\int }_{{\mathbb{R}}^n}\left|p_{\mathbf{X}}(\mathbf{x})-p_{\mathbf{Y}}(\mathbf{x})\right|\mathrm{d}\mathbf{x}.$$

$$|\mathrm{h}\left(p_{\mathbf{X}}\right)|\le$$

$$+n\log \left[2\Gamma \left(\frac{1}{\alpha }+1\right)\right]+\frac{n}{\alpha }+1.$$

$$\left|\mathrm{h}\left(p_{\mathbf{X}}\right)-\mathrm{h}\left(p_{\mathbf{Y}}\right)\right|\le$$

$$c_1{\Vert p_{\mathbf{X}}-p_{\mathbf{Y}}\Vert }_1+c_2{\Vert p_{\mathbf{X}}-p_{\mathbf{Y}}\Vert }_1\log \frac{1}{{\Vert p_{\mathbf{X}}-p_{\mathbf{Y}}\Vert }_1},$$

$$c_1=$$

$$+n\log \left[2\Gamma \left(1+\frac{1}{\alpha }\right)\right]+\frac{n}{\alpha }+1,$$

$$c_2=$$

$$p(x)=\{\begin{array}{ll}\frac{1}{x(\log x{)}^2},& x>\mathrm{e},\\ 0,& x\le \mathrm{e}.\end{array}$$

$${\int }_{\mathrm{e}}^{\infty }\frac{x^{\alpha }}{x(\log x{)}^2}\mathrm{d}x={\int }_1^{\infty }\frac{{\mathrm{e}}^{\alpha y}}{y^2}\mathrm{d}y=\infty .$$

$$\mathrm{h}\left(p\right)={\int }_{\mathrm{e}}^{\infty }\frac{1}{x(\log x{)}^2}\log (x(\log x{)}^2)\mathrm{d}x=+\infty ,$$

$$q(x)=\{\begin{array}{ll}\frac{-1}{x(\log x)(\log (-\log x){)}^2},& 0<x<{\mathrm{e}}^{-\mathrm{e}},\\ 0,& \text{otherwise}.\end{array}$$

$$\mathrm{h}\left(q\right)={\int }_0^{{\mathrm{e}}^{-\mathrm{e}}}\frac{-\log (-x(\log x)(\log (-\log x){)}^2)}{x(\log x)(\log (-\log x){)}^2}\mathrm{d}x=-\infty .$$

$$\mathrm{h}\left(\mathbf{X}\right)=\mathrm{h}\left(\mathbf{Y}\right)-\frac{1}{n}\log c_m.$$

$$q(\mathbf{y})=\frac{1}{c_m}{p}_{\mathbf{X}}\left(\frac{\mathbf{y}}{\sqrt[n]{c_m}}\right)\stackrel{a.e.}{\le }1.$$

$$q(\mathbf{y})\log \frac{1}{q(\mathbf{y})}\ge 0.$$

$${\int }_{{\mathbb{R}}^n}q(\mathbf{y})\log \frac{1}{q(\mathbf{y})}\mathrm{d}\mathbf{y}={\int }_{{\mathbb{R}}^n}q(\mathbf{y})\left|\log \frac{1}{q(\mathbf{y})}\right|\mathrm{d}\mathbf{y},$$

$${\Delta }_w={\int }_{\Vert \mathbf{y}{\Vert }_{\infty }\le w}q(\mathbf{y})\log \frac{1}{q(\mathbf{y})}\mathrm{d}\mathbf{y},$$

$$\mathrm{h}\left(q\right)=\mathrm{h}\left(\mathbf{Y}\right)=\underset{w\to \infty }{\lim }{\Delta }_w.$$

$${\Theta }_w={\int }_{\Vert \mathbf{y}{\Vert }_{\infty }\le w}q(\mathbf{y})\log \frac{q(\mathbf{y})}{{\varphi }_{n,\alpha ,c_m^{\alpha /n}v}(\mathbf{y})}\mathrm{d}\mathbf{x},$$

$${\Theta }_w=$$

$$=$$

$$+\frac{n}{c_m^{\alpha /n}v\alpha }{\int }_{\Vert \mathbf{y}{\Vert }_{\infty }\le w}\Vert \mathbf{y}{\Vert }_{\alpha }^{\alpha }q(\mathbf{y})\mathrm{d}\mathbf{y}-{\Delta }_w.$$

$${\Delta }_w=$$

$$+\frac{n}{c_m^{\alpha /n}v\alpha }{\int }_{\Vert \mathbf{y}{\Vert }_{\infty }\le w}\Vert \mathbf{y}{\Vert }_{\alpha }^{\alpha }q(\mathbf{y})\mathrm{d}\mathbf{y}-{\Theta }_w$$

$$\le$$

$${\int }_{\Vert \mathbf{y}{\Vert }_{\infty }\le w}q(\mathbf{y})\mathrm{d}\mathbf{y}\le$$

Full text

I Introduction

The differential entropy of a given probability distribution represented by the density $p(x)$ is defined as $\mathrm{h}\left({p}\right):=-\int_{\mathbb{R}}{p(x)\log p(x)\,\mathrm{d}x},$ if the integral converges (existence of entropy). First defined by Shannon in [1], the differential entropy of a random variable is derived by subtracting $\log m$ from the discrete entropy of the quantized version of the random variable with step size $1/m$, and taking its limit as $m$ tends to infinity [2, p. 247].

Assuming the existence of entropy for a convergent sequence of probability measures (in some metric), it is of importance to determine whether the differential entropies also converge (continuity of entropy). Both existence and continuity of entropy has received attention in the literature [3, 4] due to their numerous applications, e.g. see [3]. To guarantee these two properties, it is common to impose constraints on the probability density function (pdf), or alternatively, to convolve the given distribution with a normal distribution to smooth it out. For the latter approach see for instance the proof of the central limit theorem in [5], and check [4] for an application of this approach in network information theory.

In order to sketch the current landscape, we provide a non-rigorous statement of the existing results, without giving the exact technical conditions. Assume a sequence of absolutely continuous ($\mathcal{AC}$) probability measures (w.r.t. Lebesgue measure) with pdfs $p_{k}(x)$. We say that $p_{k}(x)$ converges to the pdf $p(x)$ in relative entropy if $D(p_{k}\|p)\to 0$, where $D(\cdot\|\cdot)$ is the relative entropy between the two pdfs, defined as $D(p\|q)=\int_{\mathbb{R}}{p(x)\log[{p(x)}/{q(x)}]\,\mathrm{d}x}.$ We say that the convergence holds in total variation if the total variation distance $\lVert p_{k}-p\rVert_{1}$ vanishes as $k\to\infty$, where

[TABLE]

The convergence holds in entropy if ${\mathrm{h}\left({p_{k}}\right)}\to\mathrm{h}\left({p}\right)$, and finally, the convergence holds in ratio if ${p_{k}(x)}/{p(x)}\to 1$ for all $x$ in the support of $p$. Pinsker’s inequality shows that the convergence in relative entropy implies convergence in total variation [6, p. 33]. Provided $m<p_{k}(x),\,p(x)<M$ for all $k$ and $x\in\text{supp}\{p\}$, and some positive $m,\,M$, it is shown in [7] that the convergence in relative entropy implies the convergence in entropy. Similarly, with $m<{p_{k}(x)}/{p(x)}<M$ one can show that the convergence in total variation implies the convergence in entropy. In [8], under similar conditions, it is shown that the pointwise convergence of distributions implies their convergence in entropy. In [9, 7] it is shown that the convergence in ratio accompanied with $p_{k}(x)/p(x)<M$ for some $M>0$ implies the convergence in entropy and relative entropy. A technique devised in [10, p. 12] reveals that if $p$ is the “projection” of some distribution $q$ onto a closed and convex set $\mathcal{F}$ of distributions, then, the convergence of a sequence in $\mathcal{F}$ in either entropy or total variation implies the convergence in relative entropy. With uniform point-wise convergence of both $p_{k}\to p$ and $p_{k}/p\to 1$, the convergence in entropy is obtained in [11]. Further, the condition on $p_{k}/p$ could be dropped if the distributions are of finite support. It is shown in [12] that the convergence in entropy holds if $p_{k}(x)^{1-\delta}$, $p_{k}(x)^{1+\delta}$, $p(x)^{1-\delta}$, and $p(x)^{1+\delta}$ are all integrable over the real line for some $0<\delta<1$, and the integrals of $|p_{k}(x)-p(x)|^{1+\delta}$ and $|p_{k}(x)-p(x)|^{1-\delta}$ vanish as $k\rightarrow\infty$.

Finally, a class of “regular probability distributions" for vectors of random variables is identified in [4], for which the entropy difference of any two members could be controlled by their Wasserstein distance and variance.

In this paper we define the new class $(\alpha,v,m)\text{--}\mathcal{AC}^{n}$ of $n$-dimensional probability measures for which the convergence in total variation leads to the convergence in entropy. The parameters $v$ and $m$ in this class represent upper bounds on the $\alpha$th order moment and the supremum of $p(x)$, respectively, which are easier to evaluate than the integrals of $p(x)^{1-\delta}$ and $p(x)^{1+\delta}$. This class is a subset of the one considered in [13], for which the existence (and not continuity) of the differential entropy is shown. Thus, with more stringent conditions, we prove continuity of entropy in addition to its existence. Conversely, the authors of [4] define a class of “$(c_{1},c_{2})$-regular distributions” as the set of $p(\mathbf{x})$ satisfying

[TABLE]

The above regularity condition is restrictive. For instance, any regular $p(\mathbf{x})$ needs to be differentiable, whereas $p(\mathbf{x})\in(\alpha,v,m)\text{--}\mathcal{AC}^{n}$ may not even be continuous. However, with the restrictive regularity condition, the scaling behaviour of the results in [4] in terms of $n$ are superior than those of $(\alpha,v,m)\text{--}\mathcal{AC}^{n}$: it is proved in [4, Proposition 1] that if $\mathbf{X}\sim p_{\mathbf{X}}(\mathbf{x})$ and $\mathbf{Y}\sim p_{\mathbf{Y}}(\mathbf{y})$ are $(c_{1},c_{2})$-regular, then $\mathrm{h}\left({\mathbf{X}}\right)$ and $\mathrm{h}\left({\mathbf{Y}}\right)$ exist and

[TABLE]

where $\Delta$ is a constant that grows like $\sqrt{n}$ as we increase the dimension $n$. However, the constant that we find for $(\alpha,v,m)\text{--}\mathcal{AC}^{n}$ grows like $n\log n$.

This paper is organized as follows: in Section II, some definitions are given; the main result is presented in Section III, and its proof is given in Section IV and the appendix.

II Definitions and Notations

All the logarithms in this paper are in base $\mathrm{e}$. Random variables and vectors are denoted by capital letters with vectors distinguished by bold letters.

Definition 1** (Absolutely Continuous Random Vector).**

Let $\mathcal{B}^{n}$ be the Borel $\sigma$-field of $\mathbb{R}^{n}$ and let $\mathbf{X}$ be a real-valued and $n$-dimensional random vector that is measurable with respect to $\mathcal{B}^{n}$. We call $\mathbf{X}$ an absolutely continuous random vector if its probability measure $\mu$, induced on $(\mathbb{R}^{n},\mathcal{B}^{n})$, is absolutely continuous with respect to the Lebesgue measure for $\mathcal{B}^{n}$ (i.e., $\mu(\mathcal{A})=0$ for all $\mathcal{A}\in\mathcal{B}^{n}$ with zero Lebesgue measure). We denote the set of all absolutely continuous distributions by $\mathcal{AC}$.

The Radon-Nikodym theorem implies that for each $\mathbf{X}\in\mathcal{AC}$ there exists a $\mathcal{B}^{n}$-measurable function $p:\mathbb{R}^{n}\to[0,\infty)$, such that for all $\mathcal{A}\in\mathcal{B}^{n}$ we have that

[TABLE]

The function $p$ is called the probability density function (pdf) of $\mathbf{X}$ [6, p. 21]. The property $\mathbf{X}\in\mathcal{AC}$ is alternatively written as $p\in\mathcal{AC}$. Random variables with an absolutely continuous distributions are generally known as continuous random variables.

Remark 1**.**

Intuitively speaking, the absolute continuity of a probability measure w.r.t the Lebesgue measure implies that the function has a pdf with no delta functionals, i.e., the cumulative distribution function (CDF) is continuous. It does not imply the absolute continuity (or even continuity) of its probability density function $p(\mathbf{x})$ as a function of $\mathbf{x}$.

Definition 2** (Differential Entropy).**

[2, Chapter 2]** For an $n$-dimensional $\mathbf{X}\in\mathcal{AC}$ with density $p$ we define the differential entropy $\mathrm{h}\left({\mathbf{X}}\right)$, or $\mathrm{h}\left({p}\right)$ as

[TABLE]

if the integral converges. We use the convention $x\log(x)=0$ at $x=0$.

Definition 3**.**

Given $\alpha,m,v>0$, and $n\in\mathbb{N}$, we define $(\alpha,v,m)\text{--}\mathcal{AC}^{n}$ to be the class of all $\mathbf{X}\in\mathcal{AC}$ such that the corresponding density function $p:\mathbb{R}^{n}\mapsto[0,\infty)$ satisfies

[TABLE]

where \lVert\mathbf{x}\rVert_{\alpha}:=\big{(}\sum_{i=1}^{n}{\left|x_{i}\right|^{\alpha}}\big{)}^{1/\alpha}.

Remark 2**.**

Observe that a scalar continuous random variable $X$ is in $(\alpha,m,v)-\mathcal{AC}$ if it has an $\alpha$-moment $\mathbb{E}[|X|^{\alpha}]<v$ and a bounded density function $p_{X}(x)<m$. It is often not difficult to verify these properties for a given random variable. Many commonly used random variables have bounded density functions and at least one finite moment. Furthermore, if $X_{1}\in(\alpha,m_{1},v_{1})-\mathcal{AC}$ and $X_{2}\in(\alpha,m_{2},v_{2})-\mathcal{AC}$ are independent random variables, $X_{1}+X_{2}\in(\alpha,\min(m_{1},m_{2}),c_{\alpha}(v_{1}+v_{2}))-\mathcal{AC}$ where $c_{\alpha}=1$ if $\alpha\in[0,1]$ and $c_{\alpha}=2^{\alpha-1}$ for $\alpha>1$. This is because $f_{X_{1}+X_{2}}(x)=\int_{t}f_{X_{1}}(t)f_{X_{2}}(x-t)dt\leq m_{1}\int_{t}f_{X_{2}}(x-t)dt=m_{1}$. Also, the $\alpha$-moment of $X_{1}+X_{2}$ can be bounded by the moments of $X_{1}$ and $X_{2}$ via the $c_{r}$-inequality for moments. Thus, any linear combination of the commonly used variables also belongs to the defined class of distributions.

Definition 4**.**

[14]** The density of an $n$-dimensional random vector of “$\alpha$-generalized normal distribution”, $\mathcal{N}\left(0,(v\alpha/n)^{1/\alpha}\mathbf{I}_{n},\alpha\right)$, is given by

[TABLE]

where $\mathbf{I}_{n}$ is the identity matrix of size $n$, scalars $\alpha,v>0$ are arbitrary, and $\Gamma$ is the gamma function. The $\alpha$-generalized normal distribution satisfies $\mathbb{E}\left[{\left\lVert\mathbf{X}_{n,\alpha,v}\right\rVert_{\alpha}^{\alpha}}\right]=v$.

III Main Result

Our main result stated below in Theorem 1 implies that the differential entropy is uniformly continuous over $(\alpha,v,m)\text{--}\mathcal{AC}^{n}$ with respect to the total variation metric:

[TABLE]

Theorem 1**.**

The differential entropy of any $p_{\mathbf{X}}\in(\alpha,v,m)\text{--}\mathcal{AC}^{n}$ is well-defined and satisfies

[TABLE]

Furthermore, for all $p_{\mathbf{X}},p_{\mathbf{Y}}\in(\alpha,v,m)\text{--}\mathcal{AC}^{n}$ satisfying $\left\lVert p_{\mathbf{X}}-p_{\mathbf{Y}}\right\rVert_{1}\leq m$, we have that

[TABLE]

where

[TABLE]

We now compliment the above result by showing that the differential entropy may no longer be well-defined if we remove any of (6) or (7) from the definition of $(\alpha,v,m)\text{--}\mathcal{AC}^{n}$.

Example 1**.**

Consider the following pdf defined over the real line $\mathbb{R}$:

[TABLE]

Note that $p(x)$ is a valid density function with no delta functionals; it defines a continuous random variable and thus $p\in\mathcal{AC}$. Also, observe that $p(x)<1/\mathrm{e}$ for all $x\in\mathbb{R}$. On the other hand, by applying the change of variables $y=\log x$, we observe that the $\alpha$th-order moment is infinite for any $\alpha>0$:

[TABLE]

Hence, this distribution satisfies (7), but not (6). With the same change of variables $y=\log x$, it can be observed that

[TABLE]

Example 2**.**

Consider the following probability density function:

[TABLE]

Here, (6) is satisfied but (7) is violated. Note that $q(x)$ is a valid density function with no delta functionals, thus $q\in\mathcal{AC}$. Since the support of this pdf is finite, all of its $\alpha$-th moments are finite. However, by the change of variables $y=\log(-\log x))$, we have that

[TABLE]

IV Proof of the Main Result

IV-A Proof of (10): Upper bound on the differential entropy

Let $\mathbf{Y}:=\sqrt[n]{c_{m}}\,\mathbf{X}$ where $c_{m}=\max(m,1)$. The existence of the differential entropy $\mathrm{h}\left({\mathbf{Y}}\right)$ is equivalent to the existence of the differential entropy $\mathrm{h}\left({\mathbf{X}}\right)$, and when both exist we have that [2, Theorem 8.6.4],

[TABLE]

The benefit of defining $\mathbf{Y}$ is that its density function denoted by $q$, is bounded from above by one:

[TABLE]

Hence, for any $\mathbf{y}\in\mathbb{R}^{n}$, we know that

[TABLE]

To prove the convergence of

[TABLE]

let us consider

[TABLE]

for $w>0$, where $\lVert\mathbf{y}\rVert_{\infty}:=\max_{i\in{1,\cdots,n}}{|y_{i}|}$, and $y_{i}$ is the $i$th element of $\mathbf{y}$. By recalling (21), we deduce that $\Delta_{w}$ is increasing in $w$. In addition,

[TABLE]

Thus, to prove the existence of the differential entropy $\mathrm{h}\left({\mathbf{Y}}\right)$, it suffices to find a constant $\kappa$ such that $\Delta_{w}\leq\kappa$ for all $w$. Then, $\kappa$ would also serve as an upper bound for the entropy $\mathrm{h}\left({\mathbf{Y}}\right)$. Let

[TABLE]

where $\phi_{n,\alpha,c_{m}^{\alpha/n}v}$ is defined in Definition 4. Except for the domain of the integral, $\Theta_{w}$ defines a relative entropy measure. The utility of relative entropy to bound entropy is a known technique; in particular, we link $\Theta_{w}$ to $\Delta_{w}$ by

[TABLE]

This could be rewritten as

[TABLE]

where $b_{1}:={\alpha v}/{n}\left[2\Gamma\left({1}/{\alpha}+1\right)\right]^{\alpha}$, and the last inequality is true because

[TABLE]

We now find a lower bound for $\Theta_{w}$. Note that for any $x\geq 0$ and $a>0$, we have $x\log({x}/{a})\geq x-a.$ Thus,

[TABLE]

where the validity of the last equation is coming from the fact that $\phi_{n,\alpha,c_{m}^{\alpha/n}v}$ and $q$ are both probability density functions, and their integrals over any interval is between [math] and $1$. By combining (29) and (35), we obtain that

[TABLE]

This establishes an upper bound on $\Delta_{w}$ independent of $w$, which completes the proof of the existence of $\mathrm{h}\left({\mathbf{Y}}\right)$, and in turn the existence of $\mathrm{h}\left({\mathbf{X}}\right)$.

IV-B Proof of (11): Continuity of the differential entropy

The existence of $\mathrm{h}\left({\mathbf{X}}\right)$ and $\mathrm{h}\left({\mathbf{Y}}\right)$ follow from (10). The idea is to adapt the argument of [2, Theorem 17.3.3] from discrete entropy to differential entropy. Let $\delta=\|p_{X}-p_{Y}\|_{1}$. Observe that $\delta=0$ yields $p_{\mathbf{X}}\stackrel{{\scriptstyle\rm a.e.}}{{=}}p_{\mathbf{Y}}$ and $\mathrm{h}\left({\mathbf{X}}\right)=\mathrm{h}\left({\mathbf{Y}}\right)$. For $\delta>0$, consider two new random vectors

[TABLE]

Because of Lemma 1 from the appendix, $p_{\mathbf{X}^{\prime}},p_{\mathbf{Y}^{\prime}}\in\left(\alpha,(m\mathrm{e})^{\alpha/n}v,{1}/{\mathrm{e}}\right)$–$\mathcal{AC}^{n}$, as well as,

[TABLE]

We further define a random vector $\mathbf{Z}$ with density $p_{\mathbf{Z}}$ as

[TABLE]

We shall show that $p_{\mathbf{Z}}\in\left(\alpha,2(m\mathrm{e})^{\alpha/n}/\delta,2/(\mathrm{e}\delta)\right)$–$\mathcal{AC}^{n}$. Since $p_{\mathbf{X}^{\prime}},p_{\mathbf{Y}^{\prime}}$ represent absolutely continuous distributions, the same applies to $p_{\mathbf{Z}}$. We further have

[TABLE]

Regarding the $\alpha$ moment of $\mathbf{Z}$, we can write that

[TABLE]

The latter result confirms $p_{\mathbf{Z}}\in\left(\alpha,2(m\mathrm{e})^{\alpha/n}/\delta,2/(\mathrm{e}\delta)\right)$–$\mathcal{AC}^{n}$. Now, from (10), we know that $\mathrm{h}\left({\mathbf{Z}}\right)$ exists, and

[TABLE]

This result could be simplified by applying $\delta\leq m$ as

[TABLE]

where $c_{1}$ and $c_{2}$ are given in (12), (13). Next, we show that

[TABLE]

This equation together with (39) and (46) shall complete the proof. In order to prove (47), we can write

[TABLE]

Lemma 2 in the appendix results in

[TABLE]

Consequently,

[TABLE]

which completes the proof.