How Compressible are Innovation Processes?

TL;DR

This paper introduces an entropy-based measure to evaluate the compressibility of continuous-domain innovation processes, comparing stable and Poisson processes, and providing insights into their relative compressibility.

Contribution

It defines a new entropy limit measure for innovation processes and applies it to compare the compressibility of stable and Poisson processes.

Findings

Poisson innovations are more compressible than stable innovations.

The entropy measure ranks stable innovations by tail decay.

The approach extends compressibility analysis to continuous-domain processes.

Abstract

The sparsity and compressibility of finite-dimensional signals are of great interest in fields such as compressed sensing. The notion of compressibility is also extended to infinite sequences of i.i.d. or ergodic random variables based on the observed error in their nonlinear k-term approximation. In this work, we use the entropy measure to study the compressibility of continuous-domain innovation processes (alternatively known as white noise). Specifically, we define such a measure as the entropy limit of the doubly quantized (time and amplitude) process. This provides a tool to compare the compressibility of various innovation processes. It also allows us to identify an analogue of the concept of "entropy dimension" which was originally defined by R\'enyi for random variables. Particular attention is given to stable and impulsive Poisson innovation processes. Here, our results recognize Poisson innovations as the more compressible ones with an entropy measure far below that of stable innovations. While this result departs from the previous knowledge regarding the compressibility of fat-tailed distributions, our entropy measure ranks stable innovations according to their tail decay.

Tables

Table 1. TABLE I: Summary of the notation.

Symbol	Definition
Sets	Caligraphic letters like $𝒜,𝒞,𝒟,\dots$
Real and natural numbers	$ℝ$, $\mathbb{N}$
Borel sets in $ℝ$	$ℬ\left(ℝ\right)$ or just $ℬ$
Random variables	Capital letters like: $A,X,Y,Z,\dots$
Probability density function (pdf) of (continous) $X$	$p_X$ or $q_X$ (lower-case letter $p$)
Probability mass function (pmf) of (discrete) $X$	$P_X$ (upper-case letter $P$)
Cumulative distribution function (cdf) of $X$	$F_X$
$X_0$ for a given white noise $X\left(t\right)$	${\int }_0^{1}X\left(t\right)dt$; a random variable
$\frac{1}{n}$	step-size for time quantization (time sampling)
$\frac{1}{m}$	step-size for amplitude quantization
$X_i^{\left(n\right)}$ for a given white noise $X\left(t\right)$	${\int }_{\frac{i-1}{n}}^{\frac{i}{n}}X\left(t\right)dt$; a random variable
$X_1,X_2,\cdots$	A sequence of random variables
	Distinguished from $X_i^{\left(n\right)}$ by lack of $\left(n\right)$ superscript.
${\left[x\right]}_m$	Quantization of the value of $x$ with step size $1/m$;
${\left[X_i^{\left(n\right)}\right]}_m$	Quantization of $X_i^{\left(n\right)}$ with step size $1/m$
Discrete random variable associated to	$X_D$
a discrete-continuous random variable $X$
Continuous random variable associated to	$X_c$
a discrete-continuous random variable $X$
Entropy	$H$
Differential entropy	$h$
${\mathscr{H}}_{m,n}\left(X\right)$	Quantization entropy associated to a
	white noise $X\left(t\right)$ in (11) and (12)
Characteristic function	$\hat{p}$

Equations

$$\frac{{\mathcal{H}}_{m,n}(X)}{\kappa (n)}-\log (m)-\zeta (n),$$

$${\mathcal{H}}_{m,n}(X)-\zeta (n),$$

$$\mathrm{Pr}\left\{X\in \mathcal{A}\right\}={\int }_{\mathcal{A}}p(x)\mathrm{d}x.$$

$$0<\mathrm{Pr}\left\{X\in \mathcal{D}\right\}<1,$$

$$\mathrm{Pr}\left\{X\in \mathcal{A}|X\notin \mathcal{D}\right\}={\int }_{\mathcal{A}}{p}_c(x)\mathrm{d}x,\mathrm{Pr}\left\{X\in \mathcal{A}|X\in \mathcal{D}\right\}=\sum _{x\in \mathcal{D}\cap \mathcal{A}}{P}_D[x].$$

$$p_X(x)=$$

$$\mathrm{H}\left(P_X\right)=\mathrm{H}\left(X\right):=\sum _x{P}_X[x]\log \frac{1}{P_X[x]},$$

$$\mathrm{h}\left(p_X\right)=\mathrm{h}\left(X\right):={\int }_{\mathbb{R}}{p}_X(x)\log \frac{1}{p_X(x)}\mathrm{d}x,$$

$${\int }_{\mathbb{R}}{p}_X(x)\left|\log \frac{1}{p_X(x)}\right|\mathrm{d}x<\infty .$$

$$\mathrm{H}\left(\mathbf{X}\right)=\mathbb{E}\left[\log \frac{1}{P_{\mathbf{X}}[\mathbf{X}]}\right],\text{or}\mathrm{h}\left(\mathbf{X}\right)=\mathbb{E}\left[\log \frac{1}{p_{\mathbf{X}}(\mathbf{X})}\right],$$

$${\int }_{\mathbb{R}}|x{|}^{\alpha }p(x)\mathrm{d}x\le v,$$

$$\mathrm{ess}\underset{x\in \mathbb{R}}{\sup }p(x)\le \ell .$$

$$\left|\mathrm{h}\left(p_X\right)-\mathrm{h}\left(p_Y\right)\right|\le c_1{D}_{X,Y}+c_2{D}_{X,Y}\log \frac{1}{D_{X,Y}},$$

$$D_{X,Y}={\Vert p_X-p_Y\Vert }_1:={\int }_{\mathbb{R}}\left|p_X(x)-p_Y(x)\right|\mathrm{d}x,$$

$$c_1=$$

$$c_2=$$

$${\left[X\right]}_m=\frac{1}{m}\lfloor \frac{1}{2}+mX\rfloor .$$

$$P_{X;m}[i]:=\mathrm{Pr}\left\{{{\left[{X}\right]}_{m}=\tfrac{i}{m}}\right\}=\int_{\big{[}\frac{i-0.5}{m},\frac{i+0.5}{m}\big{)}}{p_{X}(x)\,\mathrm{d}x}.$$

$$q_{X;m}(x)=m\,P_{X;m}[i],\qquad\quad x\in\big{[}\tfrac{i-0.5}{m},\tfrac{i+0.5}{m}\big{)}.$$

$$\mathrm{H}\left({\left[X\right]}_m\right)-\log m=\mathrm{h}\left({\tilde{X}}_m\right),$$

$$\mathrm{H}\left({\left[X\right]}_m\right)=$$

$$d=$$

$$h=$$

$${\mathrm{H}}_2(d)\triangleq$$

$$\mathrm{H}\left({\left[X\right]}_m\right)=\log m+\mathrm{h}\left(p_c\right)+o_m(1),$$

$$\mathrm{H}\left({\left[X\right]}_1\right)<\infty ,{\int }_{\mathbb{R}}{p}_c(x)\left|\log \frac{1}{p_c(x)}\right|\mathrm{d}x<\infty ,$$

$$\underset{t\in \mathbb{R}}{\sup }\left|t^m\frac{d^n}{d{t}^n}\varphi (t)\right|<\infty ,\forall m,n\in \mathbb{N}.$$

$$x(\phi )=⟨x,\phi ⟩={\int }_{\mathbb{R}}x(t)\phi (t)\mathrm{d}t,$$

$$\displaystyle\mathop{\text{Pr}\{X_{\varphi}\in\mathcal{I}\}}_{\mathcal{I}\in\mathcal{B}({\mathbb{R}})}=\mu\big{(}\{x\in\mathcal{S^{\prime}}(\mathbb{R})\,\big{|}\,\left\langle{x},{\varphi}\right\rangle\in\mathcal{I}\}\big{)},$$

$$\sum _{i,j=1}^n{a}_i{a}_j^{\ast }f({\omega }_i-{\omega }_j)\ge 0.$$

$$-\frac{{\sigma }^2}{2}{\omega }^2+\mathrm{j}\mu \omega +{\int }_{\mathbb{R}\setminus \{0\}}\left({\mathrm{e}}^{\mathrm{j}\omega a}-1-\mathrm{j}\omega a\text{mathds}{1}_{(-1,1)}(a)\right)\mathrm{d}V(a),$$

$$\underset{a\to -\infty }{\lim }V(a)=0,$$

Full text

I Introduction

The compressible signal models have been extensively used to represent or approximate various types of data such as audio, image and video signals. The concept of compressibility has been separately studied in information theory and signal processing societies. In information theory, this concept is usually studied via the well-known entropy measure and its variants. For instance, the notion of entropy dimension was introduced in [1] for continuous random variables based on the concept of differential entropy. The entropy dimension was later studied for discrete-domain random processes in [2] and [3] in the context of compressed sensing. In signal processing, a signal is intuitively called compressible if in its representation using a known dictionary only a few atoms contribute significantly and the rest amount to negligible contribution. Sparse signals are among the special cases for which the mentioned representation consists of a few non-zero (instead of insignificant) contributions. Compressible signals in general, and sparse signals in particular, are of fundamental importance in fields such as compressed sensing [4, 5], dimensionality reduction [6], and nonlinear approximation theory [7].

In this work, we consider the compressibility of continuous-domain innovation processes which were originally studied in the context of signal processing [8, 9]. However, we try to apply information theoretic tools to measure the compressibility. To cover the existing literature in both parts, we begin by the signal processing perspective of compressibility.

Traditionally, compressible signals are defined as infinite sequences within the Besov spaces [7], where the decay rate of the $k$-term approximation error could be efficiently controlled. A more recent deterministic approach towards modeling compressibility is via weak-$\ell_{p}$ spaces [5, 10]. The latter approach is useful in compressed sensing, where $\ell_{1}$ (or $\ell_{p}$) regularization techniques are used.

The study of stochastic models for compressibility started with identifying compressible priors. For this purpose, independent and identically distributed (i.i.d.) sequences of random variables with a given probability law are examined. Cevher in [11] defined the compressibility criterion based on the decay rate of the mean values of order statistics. A more precise definition in [12] revealed the connection between compressibility and the decay rate of the tail probability. In particular, heavy-tailed priors with infinite $p$-order moments were identified as $\ell_{p}$-compressible probability laws. It was later shown in [13] that this sufficient condition is indeed, necessary as well. A similar identification of heavy-tailed priors (with infinite variance) was obtained in [14] with a different definition of compressibility111Based on the definition in [14], a prior is called compressible if the compressed measurements (by applying a Gaussian ensemble) of a high-dimensional vector of i.i.d. values following this law could be better recovered using $\ell_{1}$ regularization techniques than the classical $\ell_{2}$ minimization approaches..

The first non-i.i.d. result appeared in [15]. By extending the techniques used in [13], and based on the notion of $\ell_{p}$-compressibility of [12], it is shown in [15] that discrete-domain stationary and ergodic processes are $\ell_{p}$-compressible if and only if the invariant distribution of the process is an $\ell_{p}$-compressible prior.

The recent framework of sparse stochastic processes introduced in [8, 9, 16] extends the discrete-domain models to continuous-domain. In practice, most of the compressible discrete-domain signals arise from discretized versions of continuous-domain physical phenomena. Thus, it might be beneficial to have continuous-domain models that result in compressible/sparse discrete-domain models for a general class of sampling strategies. Indeed, this goal is achieved in [8, 9] by considering non-Gaussian stochastic processes. The building block of these models are the innovation processes (widely known as white noise) that mimic i.i.d. sequences in continuous-domain. Unlike sequences, the probability laws of innovation processes are bound to a specific family known as infinitely divisible that includes $\alpha$-stable distributions ($\alpha=2$ corresponds to Gaussians).

The discretization of innovation processes are known to form stationary and ergodic sequences of random variables with infinitely divisible distributions. As the tail probability of all non-Gaussian infinitely divisible laws are slower than Gaussians [17, 18], they exhibit more compressible behavior than Gaussians according to [12, 15].

In this paper, we investigate the compressibility of continuous-domain stochastic processes using the quantization entropy. As a starting point, we restrict our attention in the present work to innovation processes as the building blocks of more general stochastic processes. We postpone the evaluation of the quantization entropy for more general processes to future works.

In information theory, entropy naturally arises as the proper measure of compression for Shannon’s lossless source coding problem. It also finds a geometrical interpretation as the volume of the typical sets. As the definition of entropy ignores the amplitude distribution of the involved random variables and only takes into account the distribution (or concentration) of the probability measure, it provides a fundamentally different perspective of compressibility compared to the previously studied $k$-term approximation. More precisely, the entropy reveals a universal compressibility measure that is not limited to a specific measurement technique, while the $k$-term approximation is tightly linked with the linear sampling strategy of compressed sensing. Hence, it is not surprising that our results based on the quantization entropy show that impulsive Poisson innovation processes are by far more compressible than heavy-tailed $\alpha$-stable innovation processes; the previous studies on their $k$-term approximation sort them in the opposite order when the jump distribution in the Poisson innovation is not heavy-tailed. It is interesting to mention that the same ordering of impulsive Poisson and heavy-tailed α-stable innovation processes is observed in [19].

The two main challenges that are addressed in this paper are

1. defining a quantization entropy for continuous-domain innovation processes that translates into operational lossless source coding, and 2) evaluating such a measure for particular instances to allow for their comparison. We recall that the differential entropy of a random variable $X$ with continuous range is defined by finely quantizing $X$ with resolution $1/m$, followed by canceling a diverging term $\log(m)$ from the discrete entropy of its quantized version. Obviously, we shall expect more elaborate diverging terms when dealing with continuous-domain processes. More specifically, after appropriate quantization in time and amplitude with resolutions $1/n$ and $1/m$ respectively, we propose the one of the following two expressions to cancel out the diverging terms:

[TABLE]

or

[TABLE]

where $\mathcal{H}_{m,n}(X)$ is the discrete entropy of the time/amplitude quantized process, and $\kappa(\cdot)$ and $\zeta(\cdot)$ are univariate functions. We prove that depending on the white noise process, (1) or (2) give the correct way to cancel out the diverging terms for a wide class of white Lévy noises with a suitable choice of $\kappa(n)$ and $\zeta(n)$. We may view $\kappa(n)/n$ as the analogue of entropy dimension for a white noise process. A general expression for $\kappa(n)$ is given. However, while we prove existence of a function $\zeta(n)$, we are able to provide its explicit expression only for special cases of stable and Poisson white noise processes. While the term $\log(m)$ is reminiscent of the amplitude quantization effect, functions $\kappa(\cdot)$ and $\zeta(\cdot)$ quantify the compressibility of a given Lévy process: the higher the growth rate of $\kappa(n)$, the less compressible the process. If two processes have the same growth rate of $\kappa(n)$, then, the $\zeta(n)$ with the smaller growth rate is the more compressible process.

Finally, $\epsilon$-metric entropy of [20] and $\epsilon$-entropy of [21] could be considered as other alternatives for quantifying compressibility of stochastic processes. The interested reader may refer to [22] for a general discussion of defining relative entropy for stochastic processes.

The organization of the paper is as follows. We begin by reviewing the preliminaries, including some of the basic definitions and results regarding differential entropy and white Lévy noises in Section II. Next, we present our main contributions in Section III wherein we study the quantization entropy of a wide class of white Lévy noise processes. Furthermore, special attention is given to the stable and impulsive Poisson innovation processes. To facilitate reading of the paper, we have separated the results from their proofs. Some of the key lemmas for obtaining the final claims are explained in Section IV, while the main body of proofs are postponed to Section V.

II Preliminaries

The goal of this paper is to define an quantization entropy for certain random processes. Hence, we first review the concept of entropy for random variables. For this purpose, we provide the definition of entropy for three main types of probability distributions. This is followed by the definition of innovation processes (white Lévy noises) and in particular, the stable and Poisson white noise processes.

All the logarithms in this paper are in base $\mathrm{e}$. In Table I, we summarize the notation used in this paper.

II-A Types of Random variables

The main types of random variables considered in this paper are discrete, continuous, and discrete-continuous, which are defined below.

Definition 1** (Continuous Random Variables).**

[23]** Let $\mathcal{B}$ be the Borel $\sigma$-field of $\mathbb{R}$ and let $X$ be a real-valued random variable with distribution (cdf) $F(x)$ that is measurable with respect to $\mathcal{B}$. We call $X$ a continuous random variable if its probability measure $\mu$, induced on $(\mathbb{R},\mathcal{B})$, is absolutely continuous with respect to the Lebesgue measure for $\mathcal{B}$ (i.e., $\mu(\mathcal{A})=0$ for all $\mathcal{A}\in\mathcal{B}$ with zero Lebesgue measure). We denote the set of all absolutely continuous distributions by $\mathcal{AC}$.

It is a well-known result that $X$ being a continuous random variable is equivalent to the fact that the cdf $F(x)$ is an absolutely continuous function. The Radon-Nikodym theorem implies that for each $X\sim F\in\mathcal{AC}$ there exists a $\mathcal{B}$-measurable function $p:\mathbb{R}\to[0,\infty)$, such that for all $\mathcal{A}\in\mathcal{B}$ we have that

[TABLE]

The function $p$ is called the probability density function (pdf) of $X$. The property $F\in\mathcal{AC}$ is alternatively written as $p\in\mathcal{AC}$. [22, p. 21]

Definition 2** (Discrete Random Variable).**

[23]** A random variable $X$ is called discrete if it takes values in a countable alphabet set $\mathcal{X}\subset\mathbb{R}$.

Definition 3** (Discrete-Continuous Random Variable).**

[23]** A random variable $X$ is called discrete-continuous with parameters $\left(p_{c},P_{D},\mathrm{Pr}\left\{{X\in\mathcal{D}}\right\}\right)$ if there exists a countable set $\mathcal{D}$, a discrete probability mass function $P_{D}$, whose support is $\mathcal{D}$, and a pdf $p_{c}\in\mathcal{AC}$ such that

[TABLE]

as well as for every Borel set $\mathcal{A}$ in $\mathbb{R}$ we have that

[TABLE]

It is clear that we can write the pdf of a discrete-continuous random variable $X$, $p_{X}$, as follows:

[TABLE]

where $p_{c}\in\mathcal{AC}$, and $P_{d}$ is the probability mass function of the discrete part, which is a convex combination of Dirac’s delta functions.

In this paper, the probability mass function of discrete random variables is denoted by capital letters like $P$ and $Q$, while the probability density function of continuous or discrete-continuous random variables is denoted by lowercase letters like $p$ and $q$.

II-B Definition of Entropy

We first define the entropy and differential entropy for discrete and continuous random variables, respectively. Next, we define entropy dimension for discrete-continuous random variables via amplitude quantization.

Definition 4** (Entropy and Differential Entropy).**

[24, Chapter 2]** We define entropy $\mathrm{H}\left({X}\right)$, or $\mathrm{H}\left({P_{X}}\right)$ for a discrete random variable $X$ with probability mass function (pmf) $P_{X}[x]$ as

[TABLE]

if the summation converges. For a continuous random variable $X$ with pdf $p_{X}(x)\in\mathcal{AC}$, we define differential entropy $\mathrm{h}\left({X}\right)$, or $\mathrm{h}\left({p_{X}}\right)$ as

[TABLE]

if

[TABLE]

Similarly, for a discrete or continuous random vector $\bf{X}$, the entropy or the differential entropy is defined as

[TABLE]

where $P_{\bf X}[\bf x]$ ($p_{\bf X}(\bf x)$) is the pmf (pdf) of the random vector $\bf X$, respectively.

In brief, we say that the (differential) entropy is well-defined if the corresponding (integral) summation is convergent to a finite value.

Next, we identify a class of absolutely continuous probability distributions, and show that differential entropy is uniformly convergent over this space under the total variation distance metric.

Definition 5**.**

[25]** Given $\alpha,\ell,v\in(0,\infty)$, we define $(\alpha,\ell,v)\text{--}\mathcal{AC}$ to be the class of all $p\in\mathcal{AC}$ such that the corresponding density function $p:\mathbb{R}\mapsto[0,\infty)$ satisfies

[TABLE]

Theorem 1**.**

[25]** The differential entropy of any distribution in $(\alpha,\ell,v)\text{--}\mathcal{AC}$ is well-defined, and for all $p_{X},p_{Y}\in(\alpha,\ell,v)\text{--}\mathcal{AC}$ satisfying $\left\lVert p_{X}-p_{Y}\right\rVert_{1}\leq m$, we have that

[TABLE]

where

[TABLE]

and

[TABLE]

Now, we define the quantization of a random variable in amplitude domain.

Definition 6** (Quantization of Random Variables).**

The quantized version of a random variable $X$ with the step size $1/m$ (for $m>0$) is defined as

[TABLE]

Thus, ${\left[{X}\right]}_{m}$ has the pmf $P_{X;m}$ given by

[TABLE]

Also, we define a continuous random variable $\widetilde{X}_{m}$ with pdf $q_{X;m}\in\mathcal{AC}$ as follows

[TABLE]

We state a useful lemma about the entropy of quantized random variables here:

Lemma 1**.**

[1]** Let $X\sim p_{X}(x)$ be a continuous random variable. Then,

[TABLE]

where ${\left[{X}\right]}_{m}$ is the quantization of $X$ with step size $1/m$, and $\widetilde{X}_{m}\sim q_{X;m}$ is the random variable defined in Definition 6.

The following theorem, proved in [1], measures the entropy of quantized discrete-continuous random variables by defining entropy dimension.

Theorem 2**.**

[1]** Let $X$ be a discrete-continuous random variable defined by the triplet $(p_{c},P_{D},\mathrm{Pr}\left\{{X\in\mathcal{D}}\right\})$. Let $X_{D}$ be a discrete random variable with pmf $P_{D}$ and $X_{c}$ be a continuous random variable with pdf $p_{c}$. If $\mathrm{H}\left({{\left[{X}\right]}_{1}}\right)<\infty$, $\mathrm{H}\left({X_{D}}\right)<\infty$, and $\mathrm{h}\left({X_{c}}\right)$ is well-defined and finite (i.e., $\mathbb{E}\left[{|\log(p_{X_{c}})|}\right]<\infty$), where ${\left[{X}\right]}_{1}$ is the quantized version of $X$ with step size $1$, then the entropy of quantized $X$ with step size $1/m$ can be expressed as

[TABLE]

where $o_{m}(1)$ vanishes as $m$ tends to $\infty$, and

[TABLE]

The variable $d$ is called the Entropy Dimension of $X$. The theorem is true for the discrete and continuous case with $d=\mathrm{h}\left({X_{c}}\right)=\mathrm{H}_{2}(d)=0$ and $d=1,\mathrm{H}\left({X_{D}}\right)=\mathrm{H}_{2}(d)=0$, respectively.

Remark 1**.**

[1]** According to Theorem 2, if a random variable $X$ is continuous with pdf $p_{c}$, then, the entropy of quantized $X$ with step size $1/m$ is

[TABLE]

provided that

[TABLE]

where $\mathrm{h}\left({p_{c}}\right)$ is the differential entropy of $X$.

II-C White Lévy Noises

To introduce the notion of white Lévy noises, we first define the concept of a generalized function that generalizes ordinary functions to include distributions such as Dirac’s delta function and its derivatives [16].

Definition 7** (Schwartz Space).**

[16, p. 30]** The Schwartz space, denoted as $\mathcal{S}(\mathbb{R})$, consists of infinitely differentiable functions $\phi:\mathbb{R}\mapsto\mathbb{R}$, for which

[TABLE]

In other words, $\mathcal{S}(\mathbb{R})$ is the class of smooth functions that, together with all of their derivatives, decay faster than the inverse of any polynomial at infinity.

The space of tempered distributions or alternatively, the continuous dual of the Schwartz space denoted by $\mathcal{S^{\prime}}(\mathbb{R})$, is the set of all continuous linear mappings from $\mathcal{S}(\mathbb{R})$ into $\mathbb{R}$ (also known as generalized functions). In other words, for all $x\in\mathcal{S^{\prime}}(\mathbb{R})$ and $\varphi\in\mathcal{S}(\mathbb{R})$, $x(\varphi)$ is a well-defined real number. Due to the linearity of the mapping with respect to $\varphi$, the following notations are interchangeably used:

[TABLE]

where $\left\langle{x},{\varphi}\right\rangle$, $x(t)$ and the integral on the right-hand side are merely notations. This formalism is useful because it allows for a precise mathematical definition of generalized functions, such as impulse function that are common in engineering textbooks.

Just as generalized functions extend ordinary functions, generalized stochastic processes extend ordinary stochastic processes. In particular, a generalized stochastic process is a probability measure on $\mathcal{S^{\prime}}(\mathbb{R})$. Further, observing a generalized stochastic process $X(\cdot)$ is done by applying its realizations to Schwartz functions; i.e., for a given $\varphi\in\mathcal{S}(\mathbb{R})$, $X_{\varphi}=\left\langle{X},{\varphi}\right\rangle=\int_{\mathbb{R}}{X(t)\varphi(t)\,\mathrm{d}t}$ represents a real-valued random variable with

[TABLE]

where $\mu(\cdot)$ stands for the probability measure on $\mathcal{S^{\prime}}(\mathbb{R})$ that defines the generalized stochastic process.

The white Lévy noises are a subclass of generalized stochastic processes with certain properties. Before we introduce them, we define Lévy exponents:

Definition 8** (Lévy Exponent).**

[16, p. 59]** A function $f:\mathbb{R}\to\mathbb{C}$ is called a Lévy exponent if

$f(0)=0$, 2. 2.

$f$* is continuous at [math],* 3. 3.

$\forall n\in\mathbb{N},\forall\boldsymbol{\omega}\in\mathbb{R}^{n}$, and $\forall\mathbf{a}\in\mathbb{C}^{n}$ satisfying $\sum_{i=1}^{n}{a_{i}}=0$, we have that

[TABLE]

The following theorem provides the algebraic characterization of Lévy exponents.

Theorem 3** (Lévy-Khintchin).**

[16, p. 61]** A function $f(\omega)$ is a Lévy exponent if and only if it can be written as

[TABLE]

where $\sigma,\mu\in\mathbb{R}$ are arbitrary constants. The function $\mathds{1}_{(-1,1)}(a)$ is an indicator function which is $1$ when $|a|<1$ and is [math] otherwise. The function $V(x)$ is a non-negative non-decreasing function that is continuous at $a=0$ and satisfies

[TABLE]

Definition 9** (White Lévy Noises).**

[16, Sec. 4.4]** A generalized stochastic process $X$ is called a white Lévy noise, if

[TABLE]

where $\left\langle{X},{\varphi}\right\rangle$ is the output of the linear operator $\varphi$ under $X$, and $f$ is a valid Lévy exponent for which

[TABLE]

for some $\theta>0$.

The desired properties of a white noise could be inferred from Definition 9:

Lemma 2**.**

[18]** A white Lévy noise $X$ is a stationary process in the sense that $\left\langle{X},{\varphi_{1}}\right\rangle$ and $\left\langle{X},{\varphi_{2}}\right\rangle$ have the same probability law when $\varphi_{2}(t)=\varphi_{1}(t-t_{0})$. In addition, the independent atom property of white noise could be expressed as the statistical independence of $\left\langle{X},{\varphi_{1}}\right\rangle$ and $\left\langle{X},{\varphi_{2}}\right\rangle$ when $\varphi_{1}$ and $\varphi_{2}$ have disjoint supports ($\varphi_{1}(t)\varphi_{2}(t)\equiv 0$).

Next, we explain two important types of white Lévy noises, namely stable and impulsive Poisson, that are studied in this paper.

Definition 10** (Stable random variables).**

[26, p. 5]** A random variable $X$ is stable with parameters $(\alpha,\beta,\sigma,\mu)$ if and only if its characteristic function $\widehat{p}(\omega)$ is given by

[TABLE]

where $f:\mathbb{R}\mapsto\mathbb{C}$ is

[TABLE]

$\alpha\in(0,2]$* is the stability coefficient, $\beta\in[-1,1]$ is the skewness coefficient, $\sigma\in(0,\infty)$ is the scale coefficient, and $\mu\in\mathbb{R}$ is the shift coefficient. In addition, function $\mathrm{sgn}\left({x}\right):\mathbb{R}\mapsto\{-1,0,1\}$ is the sign function defined as*

[TABLE]

Definition 11** (Stable white noise).**

[27, p. 87]** The random process $X$ is a stable innovation process with parameters $(\alpha,\beta,\sigma,\mu)$ if $X$ is a white Lévy noise with the Lévy exponent $f(\omega)$ defined in (3).

Definition 12** (Impulsive Poisson white noise).**

[16, p. 64]** When the Lévy exponent $f(\omega)$ of a white Lévy noise satisfies

[TABLE]

for some scalar $\lambda>0$ (known as the rate of impulses) and cumulative distribution function $F_{A}$ over $\mathbb{R}$ (called the amplitude cdf), then, it is called an impulsive Poisson white noise.

Lemma 3**.**

Let $X(t)$ be a white Lévy noise with parameters $\sigma=0$, $\mu$ and $V(a)$ such that

[TABLE]

Then, $X(t)$ can be decomposed as $X(t)=Y(t)+\mu^{\prime}$, where $Y(t)$ is an impulsive Poisson white noise with impulse rate $\lambda=\int_{\mathbb{R}\setminus\{0\}}{\,\mathrm{d}V(a)}<\infty$ and impulse amplitude cdf $F_{A}(a)=\frac{1}{\lambda}V(a)$. The constant $\mu^{\prime}$ is also given by

[TABLE]

The proof of the lemma can be found in Section V-H.

As can be seen from Lemma 3, the function $V$ has a great influence on the type of the random variables derived linearly from an innovation process. To clarify this fact, we consider the following decomposition on a generic $V$ to three non-negative increasing functions as [28, 1]:

[TABLE]

where

• •

the function $V_{d}$ known as the discrete part of $V$, consists of countable finite jumps at $a\in\mathbb{R}$ such that $0<V_{d}(a^{+})-V_{d}(a^{-})<\infty$, while $V_{d}(a)$ is constant elsewhere,

• •

the function $V_{ac}$ known as the absolutely continuous part of $V$, i.e. for any Borel set $\mathcal{A}$ with Lebesgue measure of [math], $\int_{\mathcal{A}}{\,\mathrm{d}V_{ac}(a)}=0$, and

• •

the function $V_{cs}$ known as the continuous singular part of $V$, is a continuous function, but not absolutely continuous.

Then, we have the following theorem:

Theorem 4**.**

Let $X(t)$ be a white noise with the triplet $(\mu,\sigma,V=V_{d}+V_{ac}+V_{sc})$, and define the random variable $X_{0}$ as

[TABLE]

Then,

[29]** $X_{0}$ is discrete if and only if

• •

$\sigma=0$,

• •

$\int_{\mathbb{R}\setminus\{0\}}{\,\mathrm{d}V(a)}<\infty$,

• •

$V_{ac}(a)\equiv V_{cs}(a)\equiv 0$. 2. 2.

[28]** $X_{0}$ is continuous if at least one of the following conditions is satisfied:

• •

$\sigma>0$,

• •

$\int_{\mathbb{R}\setminus\{0\}}{\,\mathrm{d}V_{ac}(a)}=\infty$. 3. 3.

[29]** $X_{0}$ is discrete-continuous if and only if

• •

$\sigma=0$,

• •

$\int_{\mathbb{R}\setminus\{0\}}{\,\mathrm{d}V(a)}<\infty$,

• •

$V_{cs}(a)\equiv 0$* and $V_{ac}(a)\not\equiv 0$.*

Note that the conditions for the discrete and the discrete-continuous cases are necessary and sufficient, while the conditions for the continuous case are only sufficient.

III Main Results

We first propose a criterion for the compressibility of stochastic processes, and study its operational meaning from the viewpoint of source coding. This criterion and our general results regarding its evaluation are described in Section III-A. In Section III-B, we evaluate the compressibility criterion for some special cases namely stable white noise, impulsive Poisson white noise and their sum. Finally, we present a qualitative comparison between the compressibility of the considered innovation processes in Section III-C.

III-A Compressibility via quantization

A generic continuous-domain stochastic process is spread over the continuum of time and amplitude. Hence, we doubly discretize the process by applying time and amplitude quantization. This enables us to utilize the conventional definition of the entropy measure. Then, we monitor the entropy trends as the quantizations become finer.

The amplitude quantization was previously defined in Definition 6. The time quantization, or equivalently, the sampling in time shall be defined in a similar fashion:

Definition 13** (Time Quantization).**

The time quantization with step size $1/n$ of a white Lévy noise (an innovation process) $X(t)$ is defined as the sequence \big{\{}X_{i}^{(n)}\big{\}}_{i\in\mathbb{Z}} of random variables

[TABLE]

where $\phi_{i,n}(t)=\phi\left(nt-i+1\right)$ and

[TABLE]

Remark 2**.**

Observe that $\phi(t)$ is not a member of $\mathcal{S}(\mathbb{R})$ as defined in Definition 7. Hence, strictly speaking, for a white Lévy noise $X(t)$ with sample space $\mathcal{S}^{\prime}(\mathbb{R})$, we cannot automatically define $\left\langle{X},{\phi}\right\rangle$ based on Definition 9. However, the random variables $X_{i}^{(n)}$ could be easily interpreted as the increments of the Lévy process corresponding to this white noise. Alternatively, one can define $\left\langle{X},{\phi}\right\rangle$ as the limit of $\left\langle{X},{\psi_{k}}\right\rangle$ as $k\to\infty$, when $\{\psi_{k}(t)\}_{k=1}^{\infty}\subset\mathcal{S}(\mathbb{R})$ satisfy $\lim_{k\to\infty}\int_{\mathbb{R}}{|\psi_{k}(t)-\phi(t)|\,\mathrm{d}t}=0$. For definitions and arguments in this paper, convergence in probability is sufficient for $\left\langle{X},{\psi_{k}}\right\rangle\stackbin[k\to\infty]{}{\to}\left\langle{X},{\phi}\right\rangle$ to hold.

In some cases, instead of just one function $\phi(t)$, we have multiple step functions $\phi_{i}$ for $i=0,1,\dots,n$ which are not members of $\mathcal{S}(\mathbb{R})$ and we want to simultaneously define the random variables $\left\langle{X},{\phi_{0}}\right\rangle,\dots,\left\langle{X},{\phi_{n}}\right\rangle$. To account for the simultaneous definition of $\{\left\langle{X},{\phi_{i}}\right\rangle\}_{i=0}^{n}$ that captures the joint distribution, we need the convergence in probability of the multivariate random variable $[\left\langle{X},{\psi_{k,0}}\right\rangle,\dots,\left\langle{X},{\psi_{k,n}}\right\rangle]$ to $[\left\langle{X},{\phi_{0}}\right\rangle,\dots,\left\langle{X},{\phi_{n}}\right\rangle]$, when

[TABLE]

Such convergence results could be achieved via the approach of [30].

Our next step, is to find the entropy rate of a (doubly) quantized random process. Let $X(t)$ be a white Lévy noise and define the random vectors of size $n$

[TABLE]

where $X_{i}^{(n)}$ refers to the time quantization of the process (Definition 13) followed by amplitude quantization in the form ${\left[{X_{i}^{(n)}}\right]}_{m}$ as shown in Definition 6. The time quantization in $\widetilde{\bf X}^{m,n}_{i}$ spans the interval $t\in[i-1,i)$ of the process $X(t)$ via $n$ random variables ${\left[{X_{n(i-1)+j}^{(n)}}\right]}_{m},~{}j=1,\dots,n$. Thus, the sequence \big{\{}\widetilde{\bf X}^{m,n}_{i}\big{\}}_{i\in\mathbb{Z}} represents the innovation process over the whole real axis $t\in\mathbb{R}$ in a quantized way. We evaluate the quantization entropy rate (entropy per unit interval of time) for \big{\{}{\widetilde{\bf X}^{m,n}_{i}}\big{\}}_{i\in\mathbb{Z}}

[TABLE]

where $\mathrm{H}\left({\cdot}\right)$ stands for the discrete entropy. The above definition has an operational meaning in terms of the number of bits required for asymptotic lossless compression of the source as $T$ tends to $\infty$. For fixed $m,n$ and varying $i$, since ${\left[{X_{i}^{(n)}}\right]}_{m}$’s depend on equilength and non-overlapping time intervals of the white noise, they are independent and identically distributed (Lemma 2 in Sec. II-C). Therefore,

[TABLE]

To compensate for the quantization effect, we shall study the behavior of $\mathcal{H}_{m,n}(X)$ as $m,n\to\infty$.

The following theorems consider the behavior of $\mathcal{H}_{m,n}(X)$ by showing that for a wide class of white noises we have one of the following cases

[TABLE]

for appropriate functions $m(n)$, $\kappa(n)$, and $\zeta(n)$. The following two theorems prove the existence and uniqueness of $\kappa(n)$ and $\zeta(n)$ such that (13) or (14) hold. Let us begin with the asymptotic uniqueness of $\kappa(n)$ and $\zeta(n)$ first.

Theorem 5** (Asymptotic Uniqueness of $\kappa(n)$ and $\zeta(n)$).**

Let $X(t)$ be a white Lévy noise. If one can find functions $\kappa_{i}(n)$, $\zeta_{i}(n)$, and $m_{i}(n)$ for $i=1,2$ such that

[TABLE]

then,

[TABLE]

Likewise, if

[TABLE]

then,

[TABLE]

The theorem is proved in Section V-A.

The above theorem shows that $\kappa(n)$ and $\zeta(n)$ are essentially unique, if they exist. In the following theorem, we show the existence of $\kappa(n)$ and $\zeta(n)$ under certain conditions. We further express $\kappa(n)$ in terms of the parameters of the white noise. Next, we state some facts about $\zeta(n)$ for a class of white noise processes.

Theorem 6**.**

Let $X(t)$ be a white noise with the triplet $(\mu,\sigma,V=V_{d}+V_{ac}+V_{sc})$. Assume that $X_{0}$ defined by

[TABLE]

is either a discrete, continuous, or discrete-continuous random variable (as discussed in Theorem 4). Then, the following statements hold:

• •

If $X_{0}$ is discrete:

There exist functions $m(n)$ and $\zeta(n)$ such that

[TABLE]

and

[TABLE]

for some constants $0<c_{1}\leq c_{2}<\infty$ (not depending on $n$) provided that $\mathrm{H}\left({X_{0}}\right)$ and $\mathrm{H}\left({A}\right)$ are finite, where

[TABLE]

Since $X_{0}$ is discrete $V(a)$ is discrete and bounded (Theorem 4), so the above definition of $p_{A}(x)$ corresponds to a discrete random variable $A$ and the discrete entropy $\mathrm{H}\left({A}\right)$ is meaningful.

• •

If $X_{0}$ is discrete-continuous:

Set

[TABLE]

where

[TABLE]

Then, there exist functions $m(n)$ and $\zeta(n)$ such that

[TABLE]

and

[TABLE]

for some constants $0<c_{1}\leq c_{2}<\infty$ (not depending on $n$) if random variables ${X_{0,c}}$, ${X_{0,D}}$, $X_{1,c}^{(n)}$, ${A_{c}}$, and ${A_{D}}$ defined below satisfy the technical assumptions that $\mathrm{H}\left({{\left[{X_{0}}\right]}_{1}}\right)$, $\mathrm{h}\left({X_{1,c}^{(n)}}\right)$, $\mathrm{h}\left({A_{c}}\right)$, $\mathrm{H}\left({X_{0,D}}\right)$ and $\mathrm{H}\left({A_{D}}\right)$ are well-defined for all $n$: random variables $X_{0,D}$ and $X_{0,c}$ are the discrete and continuous parts of $X_{0}$, respectively. Similarly, $X_{1,c}^{(n)}$ stands for the continuous part of $X_{1}^{(n)}$ (Theorem 4, shows that $X_{1}^{(n)}$ is also discrete-continuous if $X_{0}$ is discrete-continuous). The random variables ${A_{c}}$ and ${A_{D}}$ correspond to the distributions

[TABLE]

respectively.

• •

If $X_{0}$ is continuous:

Set $\kappa(n)=n$ and $\zeta(n)=\mathrm{h}\left({X_{1}^{(n)}}\right)$ where $X_{1}^{(n)}=\int_{0}^{\frac{1}{n}}{X(t)\,\mathrm{d}t}$. If at least one of the assumptions in part (2) of Theorem 4 is satisfied, $\mathrm{H}\left({{\left[{X_{0}}\right]}_{1}}\right)$ is finite and $\mathrm{h}\left({X_{1}^{(n)}}\right)$ is well-defined for all $n$, then, we have that

[TABLE]

Furthermore, $\zeta(n)=\mathrm{h}\left({X_{1}^{(n)}}\right)$ is a non-increasing function in $n$.

The proof can be found in Section V-B.

Remark 3**.**

In order to define ${A_{c}}$ and ${A_{D}}$, which are used in the case that $X_{0}$ is a discrete-continuous random variable, note that Theorem 4 in conjunction with Lemma 3 shows that $X_{0}$ is discrete-continuous only if the white noise is an impulsive Poisson white noise, plus a constant. Utilizing Theorem 4, we can define continuous random variable $A_{c}$ with pdf $p_{A_{c}}$ such that:

[TABLE]

where $V_{ac}$ was defined in (7). Note that, according to Theorem 4, even if $V(a)$ does not have any discrete part, $X_{0}$ can still be discrete-continuous. Hence, if the discrete part of $V(a)$ exists, one can define $A_{D}$ with pmf $p_{A_{D}}$ such that

[TABLE]

III-B Special Cases

In this section, we evaluate functions $\zeta(n)$ and $m(n)$ in cases where white noise is impulsive Poisson, stable, or sum of impulsive Poisson and stable.

Theorem 7**.**

Let $X(t)$ be a stable white noise with parameters $(\alpha,\beta,\sigma,\mu)$. We define

[TABLE]

where $\phi$ is the function defined in (10). Then, for all $m(\cdot):\mathbb{N}\to\mathbb{N}$ satisfying $\lim_{n\to\infty}\frac{m(n)}{\sqrt[\alpha]{n}}=\infty$ we have that

[TABLE]

where $\mathcal{H}_{m,n}(X)$ is defined in (12), and $\mathrm{h}\left({X_{0}}\right)$ is the differential entropy of continuous random variable $X_{0}$.

The proof is given in Section V-C.

Theorem 8**.**

Let $X(t)$ be an impulsive Poisson white noise with rate $\lambda$ and amplitude pdf $p_{A}$ such that:

[TABLE]

for some positive constants $\alpha,\ell,v$ and

[TABLE]

where $A\sim p_{A}$ and $[A]_{1}$ is the quantization of $A$ with step size one. Then, for any function $m(\cdot):\mathbb{N}\to\mathbb{N}$ satisfying $\lim_{n\rightarrow\infty}\frac{m(n)}{\log n}=\infty$ we have that

[TABLE]

where $\kappa(n)=n(1-\mathrm{e}^{-\lambda/n})$.

The proof is given in Section V-D.

Corollary 1**.**

Let $X(t)$ be a white Lévy noise with parameters $\sigma=0$, $\mu$, and absolutely continuous and bounded function $V(a)=V_{ac}(a)$, which was defined in (7). Then, utilizing Lemma 3, the compressibility of $X(t)$ is similar to the compressibility of the impulsive Poisson case with

[TABLE]

provided that $p_{A}$ satisfies the conditions in Theorem 8.

Theorem 9**.**

Let $X(t)$ be a stable white noise with parameters $(\alpha,\beta,\sigma,\mu)$ and $Y(t)$ be an impulsive Poisson white noise, independent of $X(t)$, with rate $\lambda$ and amplitude random variable $A$, such that there exists $\theta>0$ where

[TABLE]

We define

[TABLE]

where $\phi$ is the function defined in (10). Then, for $Z(t)=X(t)+Y(t)$ and all $m(\cdot):\mathbb{N}\to\mathbb{N}$ satisfying $\lim_{n\to\infty}\frac{m(n)}{\sqrt[\alpha]{n}}=\infty$ we have that

[TABLE]

where $\mathcal{H}_{m,n}(Z)$ is defined in (12).

The theorem is proved in Section V-E.

Remark 4**.**

Note that Theorem 9 assumes that the amplitude measure of the impulsive Poisson only has a finite moment $\theta$ for an arbitrary $\theta>0$, while Theorem 8 has the stronger assumption that amplitude measure is in $\left(\theta,\ell,v\right)\text{--}\mathcal{AC}$.

Corollary 2**.**

Let $Z(t)$ be a white Lévy noise with parameters $\sigma>0$, $\mu$, and bounded function $V(a)$. Then, $Z(t)$ can be decomposed as $Z(t)=X_{1}(t)+Y_{1}(t)$ where $X_{1}(t)$ is a Gaussian white noise with parameters $(0,\sigma)$ and $Y_{1}(t)$ is a white Lévy noise with parameters $\sigma=0$, $\mu$, and the bounded function $V(a)$. According to Lemma 3, we have that $Y_{1}(t)=\mu^{\prime}+Y(t)$ where $Y(t)$ is an impulsive Poisson white noise with parameters $\lambda$ and amplitude cdf $F_{A}$ where

[TABLE]

Then, the compressibility of $Z(t)$ is simillar to the above case with

[TABLE]

provided that $\mathbb{E}\left[{|A|^{\theta}}\right]<\infty$ for some $\theta>0$.

III-C Comparison and discussion

In this section, we compare different white Lévy noises based on Theorem 6. Take two white Lévy noises $X(t)$ and $Y(t)$ belonging to the general class of processes considered in Theorem 6. Let $m_{X}(n)$ and $m_{Y}(n)$ be the functions associated with these two white noises in Theorem 6. Observe that if Theorem 6 holds for a choice of $m(n)$, it will also hold for any function $m^{\prime}(n)\geq m(n)$. As a result, we can look at the sequence of pairs $(n,m)$ where $m\geq\max\{m_{X}(n),m_{Y}(n)\}$ when we compare the two white noise processes. Therefore to make the comparison we only need to compare the values of $\kappa(n)$ and $\zeta(n)$ for the two white noise processes asymptotically.

Theorem 10**.**

Let $X(t)$ and $Y(t)$ be two white Lévy noises and define $X_{0}:=\int_{0}^{1}{X(t)\,\mathrm{d}t}$ and $Y_{0}:=\int_{0}^{1}{Y(t)\,\mathrm{d}t}$. Assume that conditions in Theorem 6 hold for $X(t)$ and $Y(t)$ based on their type ($X_{0}$ and $Y_{0}$ might have different types). Let $\kappa_{X}(n),\kappa_{Y}(n)$, $\zeta_{X}(n),\zeta_{Y}(n)$ , and $m_{X}(n),m_{Y}(n)$ be the functions associated in Theorem 6 to $X$ and $Y$, respectively.

If $X_{0}$ is discrete and $Y_{0}$ is discrete-continuous, then

[TABLE]

where $m(n)$ is a function such that

[TABLE] 2. 2.

If $X_{0}$ is discrete-continuous and $Y_{0}$ is continuous, then

[TABLE]

where $m(n)$ is a function such that

[TABLE] 3. 3.

If $X_{0}$ is continuous and $Y(t)$ is a Gaussian white noise with parameters $\sigma>0$ and $V(a)=0$ for all $a\in\mathbb{R}$, then

[TABLE]

where $m(n)$ is a function such that

[TABLE]

The proof of the theorem is presented in Section V-F.

The theorem, intuitively states that when $m$ grows sufficiently faster than $n$, the discrete white noises are asymptotically more compressible than discrete-continuous noises, and the discrete-continuous white noises are more compressible than continuous noises. In addition, Gaussian white noise is the least compressible white noise.

In the following theorem, we compare impulsive Poisson and stable white noises amongst themselves.

Theorem 11**.**

The following statements apply to impulsive Poisson and stable white noises.

1. i)

A stable white noise becomes less compressible as its stability parameter $\alpha$ increases. Let $X_{1}$ and $X_{2}$ be stable white noises with stability parameters $0<\alpha_{1}<\alpha_{2}\leq 2$, and functions $m_{1}(n)$ and $m_{2}(n)$ defined in Theorem 6, respectively. Then, for $m(n)\geq\max\{m_{1}(n),m_{2}(n)\}$, we have that

[TABLE]

[TABLE] 2. ii)

An impulsive Poisson white noise becomes less compressible as its rate parameter $\lambda$ increases. Let $X_{1}$ and $X_{2}$ be impulsive Poisson white noises with rate parameters $0<\lambda_{1}<\lambda_{2}$, absolutely continuous amplitude distributions, and functions $m_{1}(n)$ and $m_{2}(n)$ defined in Theorem 6, respectively. Then, for $m(n)\geq\max\{m_{1}(n),m_{2}(n)\}$, we have that

[TABLE]

[TABLE]

The proof can be found in Section V-G.

We should mention that Theorem 11 confirms with previously studied notions of compressibility in [12, 14, 15]. However, Part 2 of Theorem 10 deviates from the available literature by identifying the impulsive Poisson white noises as more compressible than heavy-tailed stable white noises. This difference is fundamental and is caused by basing the compressibility definition on the probability concentration properties of processes rather than their amplitude distribution.

IV Some Useful Lemmas

In Section IV-A, we provide a useful theorem for Poisson white noises, and in Section IV-B, we give some lemmas about quantization of a random variable in amplitude domain. These results are used in the proof of the main results.

IV-A Poisson and Stable White Noises

The following theorem states a feature of integrated Poisson white noise in a small interval.

Lemma 4**.**

Let $X(t)$ be an impulsive Poisson white noise with rate $\lambda$ and amplitude pdf $p_{A}\in\mathcal{AC}$. Define $Y_{n}$ as

[TABLE]

Then, we have that

[TABLE]

[TABLE]

where $p_{Y_{n}}$ is the probability density function of $Y_{n}$, $*$ is the convolution operator, and $A_{n}$ is a random variable with probability density function $p_{A_{n}}$. Further, we have that

[TABLE]

[TABLE]

where $A$ is a random variable with the probability density function $p_{A}$. Also, the upper bound on the total variation distance in (27) vanishes as $n$ tends to infinity.

The main novelties of Lemma 4 are (27) and (28), as (25) and (26) are rather classical results. To make the paper self-contained, we prove all the identities in Section V-I.

Lemma 5**.**

Let $X$ be a stable white noise with parameters $(\alpha,\beta,\sigma,\mu)$. Define random variable $X_{0}$ with pdf $p_{X_{0}}$ as follows:

[TABLE]

where $\phi$ is the function defined in (10). Then,

[TABLE]

[TABLE]

[TABLE]

[TABLE]

Moreover, we have that

[TABLE]

Except (33), other claims in Lemma 5 are known results in the literature. Again, we include the proof of all parts in Section V-J for the sake of completeness.

Lemma 6**.**

For a stable white noise with parameters $(\alpha,\beta,\sigma,\mu)$ we have that

[TABLE]

where $X_{1}^{(n)}$ was defined in Definition 13,

[TABLE]

and

[TABLE]

The proof can be found in Section V-K

IV-B Amplitude Quantization

In the following lemma, we show that the total variation distance between two variables decreases by quantizing; furthermore, moments of a quantized random variable tends to the moments of the original random variable, as the quantization step size vanishes.

Lemma 7**.**

Let random variables $X\sim p_{X}$ and $Y\sim p_{Y}$ be continuous, and ${\left[{X}\right]}_{m}\sim P_{X;m}$ and ${\left[{Y}\right]}_{m}\sim P_{Y;m}$ be their quantized version, defined in Definition 6, respectively. Then for all $m\in(0,\infty)$ we have

[TABLE]

where $\widetilde{X}_{m}\sim q_{X;m}$ and $\widetilde{Y}_{m}\sim q_{Y;m}$ are random variables defined in Definition 6. In addition, for any $\alpha\in(0,\infty)$, and $m\geq 4$ we have

[TABLE]

[TABLE]

provided that $\mathbb{E}\left[{|X|^{\alpha}}\right]$ exists.

The proof is given in Section V-L.

Corollary 3**.**

Let $X\sim p_{X}$ be a continuous random variable, and let $\widetilde{X}_{m}\sim q_{X;m}$ be the random variable defined in Definition 6. Then, we have that

[TABLE]

if $\mathbb{E}\left[{|X|^{\alpha}}\right]$ exists.

Proof.

Since $p_{X}\in\mathcal{AC}$, we conclude that $\mathrm{Pr}\left\{{|X|\leq{1}/{\sqrt{m}}}\right\}$ vanishes as $m$ tends to $\infty$. Hence, the corollary achieved from Lemma 7. ∎

The following lemma discusses the entropy of quantized version of multiplies of a continuous random variable.

Lemma 8**.**

Let $X\sim p$ be a continuous random variable, and $m\in(0,\infty)$ be arbitrary. If $\mathrm{H}\left({{\left[{X}\right]}_{m}}\right)$ exists, then for all $a\in(0,\infty)$, we have

[TABLE]

The lemma is proved in Section V-M.

In the following lemma, we extend Remark 1 to an arbitrary shifted continuous random variables.

Lemma 9**.**

Let $X\sim p$ be a continuous random variable with a piecewise continuous pdf $p(x)$. For an arbitrary sequence $\{c_{m}\}_{m=1}^{\infty}\subset\mathbb{R}$, we have

[TABLE]

provided that

[TABLE]

where ${\left[{X}\right]}_{1}$ is the quantized version of $X$ with step size $1$.

The proof can be found in Section V-N.

IV-C On Sum of Independent Random Variables

The following lemma is well-known and can be easily proved. Hence, we only mention it without proof.

Lemma 10**.**

Let $X,Y$ be two independent discrete-continuous random variables where $p,q$ are the probability of $X,Y$ being discrete, respectively. Hence, we can write

[TABLE]

where $X_{D},Y_{D}$ are discrete, and $X_{c},Y_{c}$ are continuous random variables. Therefore, $Z=X+Y$ is also a discrete-continuous random variable such that

[TABLE]

where the first case makes the discrete part while the other cases make the continuous part of $Z$.

Lemma 11**.**

Let $X,Y$ be two independent discrete-continuous random variables where $X_{D},Y_{D}$ are the discrete part, and $X_{c},Y_{c}$ are the continuous part of $X$ and $Y$, respectively. Then, we have that

[TABLE]

where $Z=X+Y$.

The proof can be found in Section V-O.

Corollary 4**.**

Using induction, it can be proved that for i.i.d. continuous-discrete random variables $X_{1},\cdots,X_{n}$, we have that

[TABLE]

where $S_{n,c}$ and $X_{1,c}$ are the corresponding continuous random variables of $X_{1}+\cdots+X_{n}$ and $X_{1}$, respectively.

Lemma 12**.**

Let $X$ be a continuous random variable defined in Definition 1 with probability density function $p_{X}(x)$ and $Y$ be an arbitrary random variable with probability measure $\mu_{Y}$, independent of $X$. Let $Z=X+Y$ with probability measure $\mu_{Z}$. Then, $Z$ is also continuous random variable with pdf $p_{Z}(z)$ defined as following:

[TABLE]

Moreover,

[TABLE]

provided that

[TABLE]

The theorem is proved in Section V-P.

IV-D Maximum of $\zeta(n)$ for a Class of White Noise Processes

In the following lemma we find maximum $\zeta(n)$, defined in Theorem 6 over a class of white noise processes.

Lemma 13**.**

Let $X(t)$ be a white Lévy noise such that $X_{0}:=\int_{0}^{1}{X(t)\,\mathrm{d}t}$ is a continuous random variable (as discussed in Theorem 4). Furthermore, assume that $h(X_{0})$ and $\mathrm{H}\left({{\left[{X_{0}}\right]}_{1}}\right)$ are well-defined. Let $m^{\prime}(n)$ and $\zeta(n)$ be any function satisfying the statement (17) in Theorem 6 for the white noise $X(t)$. Define $m(n)$ an arbitrary function that $m(n)\geq m^{\prime}(n)$ and $\lim_{n\to\infty}{m(n)/\sqrt{n}}=\infty$. Then, we have that

[TABLE]

The theorem is proved in Section V-Q.

V Proofs

V-A Proof of Theorem 5

Proof of the first case:

Proof of (15): There is $n_{0}\in\mathbb{N}$ such that for any $n\geq n_{0}$ we have

[TABLE]

where $m(n)$ can be any function larger than $\max\{m_{1}(n),m_{2}(n)\}$. We can write

[TABLE]

Hence, utilizing (37) for $i=2$, we have that for all $m\geq m(n)$, we have that

[TABLE]

If $\kappa_{1}(n)\neq\kappa_{2}(n)$, we can select $m>m(n)$ large enough such that the right-hand side of the above equation becomes larger than $1$. Therefore, (37) cannot be satisfied for $i=1$. Thus $\kappa_{1}(n)=\kappa_{2}(n)$ for any $n\geq n_{0}$.

Proof of (16): From (15) we obtain that for large enough $n$ we have that $\kappa_{1}(n)=\kappa_{2}(n)$. Hence, for large enough $n$, we can write

[TABLE]

where $m(n):=\max\{m_{1}(n),m_{2}(n)\}$. Therefore, according to the assumption, we obtain that

[TABLE]

Thus, the statement was proved.

Proof of the second case: For this case, similarly, we can write

[TABLE]

Therefore the statement is proved. ∎

V-B Proof of Theorem 6

From (12) we obtain that

[TABLE]

Now, we are going to use Theorem 2 in order to show that

[TABLE]

where $d_{n}$ is the entropy dimension of $X_{1}^{(n)}$ and $e_{m,n}$ vanishes for every fixed $n$ when $m$ tends to $\infty$. To this end, first, we prove the conditions of Theorem 2 which are the same for all types of discrete, continuous and discrete-continuous:

$X_{1}^{(n)}$ is a discrete, continuous, or discrete-continuous random variable, and has the same type as $X_{0}$, 2. 2.

$\mathrm{H}\left({{\left[{X_{1}^{(n)}}\right]}_{1}}\right)$ is finite.

In the next step, we prove the exclusive conditions for each case. By choosing $\kappa(n)=1$ in the discrete or $\kappa(n)=nd_{n}$ in other two cases, and $\zeta(n)=nh_{n}/\kappa(n)$, we only need to find $m(n)$ such that

[TABLE]

Since $e_{m,n}$ vanishes as $m$ tends to $\infty$ for any fixed $n$, there is some $m(n)$ such that for any $m\geq m(n)$ we have

[TABLE]

This proves the existence of the function $m(n)$.

It remains to prove items 1 and 2, and then, find $\kappa(n)$ and $\zeta(n)$ for cases of $X_{0}$ being discrete, continuous, or discrete-continuous.

Proof of item 1: Random variable $X_{1}^{(n)}$ is the time quantization with step size $1/n$ of a white Lévy noise with parameters $\sigma$, $\mu$ and $V(a)$. It can be alternatively expressed as the time quantization with unit step size of a new white Lévy noise with scaled parameters $\sigma/\sqrt{n}$, $\mu/n$ and $V(a)/n$. Theorem 4 gives conditions for $X_{0}$ (time quantization with unit step size) being discrete, continuous, or discrete-continuous random variable in terms of $\sigma$, $\mu$, and $V(a)$. These conditions for $\sigma$, $\mu$, and $V(a)$ are equivalent for the conditions for $\sigma/\sqrt{n}$, $\mu/n$, and $V(a)/n$. Therefore, random variable $X_{1}^{(n)}$ has the same type as $X_{0}$.

Proof of item 2: From Definition 9, observe that $X_{0}$ can be written as the sum of $n$ iid random variables $X_{i}^{(n)}$ for $i\in\{1,\cdots,n\}$. Hence, we have that

[TABLE]

Observe that for any quantization of sum of variables we can write

[TABLE]

where $E_{n}$ is a random variable taking values from $\{0,\cdots,n-1\}$. Observe that

[TABLE]

Therefore, $\mathrm{H}\left({{\left[{X_{1}^{(n)}}\right]}_{1}}\right)<\infty$.

Now, we aim to check the exclusive constraints and find $\kappa(n)$ and $\zeta(n)$ for each case.

Case 1: $X_{0}$ is continuous and at least one of the assumptions in part (2) of Theorem 4 is satisfied: For the first claim, as it was proved in item 1 at the beginning of the proof, observe that $X_{1}^{(n)}$ is continuous. Therefore, according to Theorem 2, $d_{n}$ is $1$ for every $n$, and

[TABLE]

provided that $\mathrm{h}\left({X_{1}^{(n)}}\right)$ is well-defined, which is true based on the assumption.

Based on Definitions 9 and 13, we can write

[TABLE]

where $E$ and $X_{1}^{(n)}$ are independent and

[TABLE]

Now observe that for any two independent continuous random variables $X,Y$ we have that

[TABLE]

Therefore,

[TABLE]

As a result, $\zeta(n)$ is a non-increasing function.

Case 2: $X_{0}$ is discrete-continuous: From item 1 from the beginning of the proof, $X_{1}^{(n)}$ is discrete-continuous if $X_{0}$ is discrete-continuous. From Lemma 3 we have that $X_{1}^{(n)}$ is sum of a constant $\mu^{\prime}/n:=(\mu/n)-(\lambda/n)\int_{[-1,1]\setminus\{0\}}{a\,\mathrm{d}V(a)}$ and the integral over $[0,\frac{1}{n}]$ of an impulsive Poisson white noise with rate $\lambda$ and discrete-continuous amplitude density $p_{A}(x)$ where

[TABLE]

The reason that $A$ is discrete-continuous is that, according to Theorem 4, when $X_{0}$ is discrete-continuous, the function $V(a)$ must be bounded and discrete-continuous i.e. $V_{cs}(a)\equiv 0$. Note that, according to Remark 3, the discrete and the continuous parts of $A$ are $A_{D}$ and $A_{c}$ with distributions $p_{A_{D}}$ and $p_{A_{c}}$, respectively.

The distribution of the integral of an impulsive Poisson white noise over $[0,\frac{1}{n}]$ is given in Lemma 4. Consider a Poisson random variable $Q_{1}$ with rate $\lambda/n$. Then, utilizing Lemma 4, we can define the following conditional distribution:

[TABLE]

According to Theorem 2, we have that

[TABLE]

Therefore, by the definition of $Q$ we can write

[TABLE]

where $A_{i}$ are iid random variable with probability density $p_{A}(x)$. Utilizing the result of Lemma 10 that

[TABLE]

where $X,Y$ are independent discrete-continuous random variables, we have that

[TABLE]

Therefore, the problem of $\kappa(n)$ is settled.

In order to find $\zeta(n)$, we apply Theorem 2 to $X_{1}^{(n)}$. Let $X_{c}^{(n)}$ and $X_{D}^{(n)}$ be the continuous and the discrete part of $X_{1}^{(n)}$, respectively. Then, to apply Theorem 2, we need to show that $|\mathrm{h}\left({X_{c}^{(n)}}\right)|$ and $\mathrm{H}\left({X_{D}^{(n)}}\right)$ are well-defined. Then, Theorem 2 implies that

[TABLE]

Since $d_{n}$ given in (40) vanishes as $n$ goes to infinity, to show the claim of the theorem for this part, it suffices to prove that

[TABLE]

for some constants $0\leq c_{3},c_{4}<\infty$ and $0<c_{5}\leq c_{6}<\infty$ not depending on $n$. Observe that (41) and (42) imply that $|\mathrm{h}\left({X_{c}^{(n)}}\right)|$ and $\mathrm{H}\left({X_{D}^{(n)}}\right)$ are well-defined. Thus, the conditions of Theorem 2 are satisfied if we can show (41)-(43). It remains to show (41)-(43).

Proof of (41): From Definition 9, for $X_{0}:=\int_{0}^{1}{X(t)\,\mathrm{d}t}$, we have that $X_{0}=\sum_{i=1}^{n}{X_{i}^{(n)}}$, where $X_{i}^{(n)}$ are i.i.d.. We denote the discrete and continuous part of $X_{0}$ and $X_{1}^{(n)}$ with $X_{0,D},X_{0,c}$ and $X_{1,D}^{(n)},X_{1,c}^{(n)}$, respectively. Therefore, according to Corollary 4 we have that

[TABLE]

Hence, if $\mathrm{h}\left({X_{0,c}}\right)<\infty$, then $\mathrm{h}\left({X_{1,c}^{(n)}}\right)<\infty$ uniformly on $n$. Note that based on the assumption, $\mathrm{h}\left({X_{1,c}^{(n)}}\right)$ well-defined.

Now, it only remains to prove $\mathrm{h}\left({X_{1,c}^{(n)}}\right)>-\infty$. To this end, similar to what we did in (39). Let

[TABLE]

where $A_{i}$ are iid random variable with probability density $p_{A}(x)$. Let $(S_{k})_{c}$ be the continuous part of $S_{k}$. Utilizing the definitions given in (38), Lemma 4 and Lemma 10, one can define random variable $Q_{2}$ as follows:

[TABLE]

[TABLE]

Hence, we can write

[TABLE]

Now, from Corollary 4, we obtain that

[TABLE]

where $A_{c}$ is the continuous part of $A$. Therefore, if $\mathrm{h}\left({A_{c}}\right)>-\infty$, the statement is proved.

Proof of (42): Similar to what we did above, utilizing the definitions given in (38), Lemma 4 and Lemma 10, one can define random variable $Q_{3}$ as following:

[TABLE]

[TABLE]

where $(S_{k})_{D}$ is the discrete part of $S_{k}$ defined in (44) for $A_{i}$ begin iid random variable with probability density $p_{A}(x)$. If we let $(A_{i})_{D}$ to be the discrete part of $A_{i}$, Lemma 4 shows that

[TABLE]

We can write that

[TABLE]

Now, in order to prove (42), it suffices to show that

[TABLE]

for some constants $c_{7}$ and $c_{8}$.

For the first inequality, since

[TABLE]

using (45) we have

[TABLE]

It can be verified that $n\mathbb{E}\left[{Q_{3}}\right]$ converges to $\lambda(1-\mathrm{Pr}\left\{{A\text{ is discrete}}\right\})$ as $n$ tends to infinity. As a result, $n\mathrm{H}\left({X_{1,D}^{(n)}|Q}\right)$ is bounded uniformly on $n$.

For the second inequality, note that among all the discrete distributions defined on $\{0,1,\cdots\}$ with a specified mean $\mu$, we have that[24, Theorem 13.5.4]:

[TABLE]

Therefore,

[TABLE]

It suffices to prove that

[TABLE]

for some positive constants $c_{9}$ and $c_{10}$. The reason is that the above equation shows existence of $n_{0}$ such that for $n>n_{0}$,

[TABLE]

Then, we have that

[TABLE]

It only remains to prove (46). To this end, we calculate $\mathbb{E}\left[{Q_{3}}\right]$ as following:

[TABLE]

As a result, for large enough $n$ there exists global constant $0<c_{9}\leq c_{10}<\infty$ such that (46) is satisfied.

Proof of (43): It can be proved that

[TABLE]

Therefore, (43) is proved.

Case 3: $X_{0}$ is discrete: As it was proved in item 1, observe that $X_{1}^{(n)}$ is discrete if $X_{0}$ is discrete. Therefore, from Theorem 2, $d_{n}$ is [math] for every $n$, and there is no term of $\log m$ in $\mathcal{H}_{m,n}(X)$. However, regardless of the results of Theorem 2, we choose $\kappa(n)=1$, and,

[TABLE]

From Theorem 4, we obtain that if $X_{0}$ is discrete, then $V(a)=V_{d}(a)$ and is bounded. Hence, according to Lemma 3, $X_{1}^{(n)}$ is sum of the integral of a Poisson impulsive white noise, plus a constant. Therefore, the calculation of finding $\zeta(n)$ is completely similar to (42) in the discrete-continuous case. Thus, the theorem is proved. ∎

V-C Proof of Theorem 7

From Lemma 6 we obtain that

[TABLE]

where $b_{n}$ is defined in Lemma 6. Hence,

[TABLE]

Therefore, from Lemma 8 we obtain that

[TABLE]

Lemma 5 shows that the distribution of $X_{0}$ satisfies the properties of Lemma 9. Thus, from Lemma 9 we obtain that when $m\geq m(n)$ and $\frac{m(n)}{\sqrt[\alpha]{n}}$ tends to $\infty$, it can be shown that

[TABLE]

Statement of the theorem follows from (47)-(49). ∎

V-D Proof of Theorem 8

From Lemma 3 we obtain that, for an impulsive Poisson white noise $X(t)$, $X_{0}:=\int_{0}^{1}{X(t)\,\mathrm{d}t}$ is a discrete-continuous random variable; as a result, utilizing Theorem 4, $X_{1}^{(n)}$ is also discrete-continuous. In order to find the probability of $X_{1}^{(n)}$ being continouous, observe that, in (25) in Lemma 4, $p_{A_{n}}$ is an absolutely continuous distribution due to (26), the fact that $p_{A}$ is absolutely continuous, and Lemma 10. Hence, from Theorem 6, we obtain that $\mathrm{Pr}\left\{{X_{1}^{(n)}\text{ is discrete}=\mathrm{e}^{-\frac{\lambda}{n}}}\right\}$, thereby

[TABLE]

In order to find $\zeta(n)$ and $m(n)$ we need to write $\mathcal{H}_{m,n}(X)$ as following. Let $P_{X_{1}^{(n)};m}$ be the pmf of ${\left[{X_{1}^{(n)}}\right]}_{m}$ as in Definition 6. Therefore,

[TABLE]

Proving the following limits will imply the first statement of the theorem:

[TABLE]

[TABLE]

where $\mathrm{h}\left({A}\right)$ is the differential entropy of random variable $A$. Proving the above equalities, will imply the statement of the theorem.

Proof of (51): Observe that from (50) we have that

[TABLE]

Therefore, it is enough to only prove

[TABLE]

From Lemma 4, we obtain that

[TABLE]

for some random variable $A_{n}$ with features mentioned in (26)-(28). For every $m$ we have

[TABLE]

Hence,

[TABLE]

where (56) is true due to (27), and the fact that $p_{A}(x)\leq M$ almost everywhere. Thus, we obtain that

[TABLE]

Therefore, from the definition of $P_{A_{n};m}[0]$ defined in Definition 6, we obtain that

[TABLE]

Therefore, from (55), we can write that

[TABLE]

Hence, if we prove that

[TABLE]

we can conclude from (58) that

[TABLE]

This will complete the proof of (54). Thus, it only remains to prove (59). Again, by substituting the value of $P_{X_{1};m}[0]$ from (55), we have

[TABLE]

where (60) follows from Taylor series of $\log(1+x)$ near [math] for $x=(\mathrm{e}^{-{\lambda}/{n}}-1)\left(1-P_{A_{n};m}[0]\right)$; observe that $\mathrm{e}^{-{\lambda}/{n}}-1$ is close to zero for large values of $n$, and (61) is obtained from (57).

Proof of (52): From Lemma 4 we obtain that

[TABLE]

By substituting the value of $P_{X_{1}^{(n)};m}[i]$ in terms of $P_{A_{n};m}[i]$, we have

[TABLE]

Therefore, in order to prove (52), utilizing (50), and it suffices to show that

[TABLE]

[TABLE]

[TABLE]

Proof of (62): It is obtained from (57).

Proof of (63): Let us add and subtract the term $\left(1-P_{A_{n};m}[0]\right)\log n$ inside of the absolute value of the left hand side of (63). Hence, we have

[TABLE]

From (57), and (53), we can write

[TABLE]

Hence, we only need to show that

[TABLE]

For (65), from (56), we have that

[TABLE]

Thus, (65) is proved due to the choice of $m(n)$, which causes $(\log n)/{m}$ tends to [math], and the fact that

[TABLE]

Proof of (64): Remember that in Definition 6, for every arbitrary random variable $X$, a random variable $\widetilde{X}_{m}$ with an absolutely continuous distribution $q_{X;m}(x)$ was defined. Thus, corresponding to $A_{n}$, we can define random variables $\widetilde{A}_{nm}$ with an absolutely continuous distribution $q_{A_{n};m}(x)$. Observe that, from Lemma 1,

[TABLE]

where $\widetilde{A}_{nm}$ is a continuous random variable with pdf $q_{A_{n};m}(x)$. Similarly, for random variable $A$, using Definition 6, we can define continuous random variable $\widetilde{A}_{m}$ with pdf $q_{A;m}(x)$. Again, from Lemma 1

[TABLE]

From (68) and Remark 1, we obtain

[TABLE]

Hence, from (67) and (69), we obtain that in order to prove (64), it suffices to show that

[TABLE]

To show this, we utilize Theorem 1. This theorem reduces convergence in differential entropy to convergence in total variation distance for a restricted class of distributions. In other words, to show (70), it suffices to show

[TABLE]

as long as we can show that $q_{A_{n};m}(x)$ and $q_{A;m}(x)$ for all $m,n\in\mathbb{N}$ belong to the class of distributions given in Definition 5. Remember that in the statement of the theorem, in equation (19), we had assumed that

[TABLE]

for some positive values for $\alpha,\ell,v$. We show that for all $m,n\in\mathbb{N}$

[TABLE]

where $v^{\prime}=2v+3$. In other words, for all $m,n\in\mathbb{N}$ we have

[TABLE]

[TABLE]

[TABLE]

As a result, it remains to show (71), (73), (74) and (75).

Proof of (71): From (34) in Lemma 7, for all $m,n\in\mathbb{N}$ we have

[TABLE]

Thus,

[TABLE]

where (76) follows from Lemma 4. Hence, (71) is proved.

Proof of (73): Since the pdfs $q_{A_{n};m}$ and $q_{A;m}$ are combination of step functions, hence, their cdf are absolutely continuous.

Proof of (74): Since there exists some $\ell$ such that $p_{A}(x)\leq\ell$ for almost all $x\in\mathbb{R}$, we obtain that

[TABLE]

It remains to show that $q_{A_{n};m}(x)$ is bounded too. Similar to the above equation, it suffices to show that $p_{A_{n}}(x)\leq\ell$. From (26) in Lemma 4, we have that

[TABLE]

where (77) follows from the fact that $p_{A}$ is bounded, and and Lemma 12.

Proof of (75): From (35) in Lemma 7, we obtain that there exists $M_{1}\in\mathbb{N}$ such that for all $m>M_{1}$ and $n\in\mathbb{N}$ we have

[TABLE]

[TABLE]

From (78) we obtain that

[TABLE]

From (28) in Lemma 4 we obtain that there exists $N_{1}\in\mathbb{N}$ such that for all $n>N_{1}$ we have

[TABLE]

Therefore, due to (79) and (81), we obtain that

[TABLE]

Thus, (75) is proved. ∎

V-E Proof of Theorem 9

From (12) and using the fact that $Z(t)=X(t)+Y(t)$, we obtain that

[TABLE]

Therefore, we aim to prove that

[TABLE]

In the proof of Theorem 7, we compute the quantization entropy of a stable white noise process by showing that

[TABLE]

Here, we essentially want to prove that we can ignore $Y_{1}^{(n)}$ in the sum $X_{1}^{(n)}+Y_{1}^{(n)}$ when we are computing the quantization entropy. To show the theorem, we only need to prove that

[TABLE]

From Lemma 6 we obtain that

[TABLE]

where $b_{n}$ is defined in Lemma 6. Hence,

[TABLE]

where the last equality follows from Lemma 8. Similarly,

[TABLE]

For some technical reason that will be needed in the proof, we want to show that $-b_{n}$ in the (85) and (86) can be replaced by a number $r_{n}<\frac{\sqrt[\alpha]{n}}{m}$ without changing the value of the entropies. Write $-b_{n}$ as

[TABLE]

for some $k\in\mathbb{Z}$ and

[TABLE]

Then,

[TABLE]

Similarly, we have

[TABLE]

Let $V_{n}:=X_{0}+r_{n}$ and $W_{n}:=V_{n}+\sqrt[\alpha]{n}Y_{1}^{(n)}$. Then, to prove (84), we only need to show that

[TABLE]

From Lemma 5, and Lemma 12 it can be obtained that $V_{n},W_{n}$ are both continuous random variables with pdfs $q_{V_{n}},q_{W_{n}}$, respectively.

Define $\widetilde{V}_{n,m/\sqrt[\alpha]{n}}\sim q_{V_{n};m/\sqrt[\alpha]{n}}$ and $\widetilde{W}_{n,m/\sqrt[\alpha]{n}}\sim q_{W_{n};m/\sqrt[\alpha]{n}}$ according to Definition 6 for random variables $V_{n}$ and $W_{n}$, respectively. From Lemma 1, we have that

[TABLE]

Therefore,

[TABLE]

To show (88), we need to show that the difference $\mathrm{h}\left({q_{V_{n};m/\sqrt[\alpha]{n}}}\right)-\mathrm{h}\left({q_{W_{n};m/\sqrt[\alpha]{n}}}\right)$ vanishes in the limit. To show this, we utilize Theorem 1: we prove existence of constants $0<\gamma,\ell,v<\infty$ such that for any $m$ and $n$ satisfying $m/\sqrt[\alpha]{n}\geq 1$ we have

[TABLE]

[TABLE]

Then, Theorem 1 yields

[TABLE]

for some constants $c_{1}$ and $c_{2}$ (not depending on $m$ and $n$) and $\Delta=2\left(1-\mathrm{e}^{-\frac{\lambda}{n}}\right)$. The differential entropy difference in (93) vanishes as $\Delta$ vanishes when $n$ goes to infinity. This completes the proof. It only remains to prove (91) and (92).

Proof of (91): Lemma 7 shows that it is enough to prove that

[TABLE]

From (25) in Lemma 4 we obtain that $\sqrt[\alpha]{n}Y_{1}^{(n)}=W_{n}-V_{n}$ has the following pdf:

[TABLE]

where

[TABLE]

is absolutely continuous. Therefore, $p_{W_{n}}$ is

[TABLE]

Then, in order to find the total variation, we can write

[TABLE]

Hence, we can write that

[TABLE]

Proof of (92): From Definition 6, it is clear that $\widetilde{V}_{n,m/\sqrt[\alpha]{n}},\widetilde{W}_{n,m/\sqrt[\alpha]{n}}$ have density and are continuous. It remains to prove the other requirements:

• •

We show that there exists $\ell<\infty$ that $q_{V_{n};m/\sqrt[\alpha]{n}}(x),q_{W_{n};m/\sqrt[\alpha]{n}}(x)\leq\ell$ for all $m,n$ and $x\in\mathbb{R}$.

Lemma 5 shows that $X_{0}$ has a density which is bounded by some $\ell<\infty$. The same bound applies to the density of $V_{n}:=X_{0}+r_{n}$, which is a shifted version of $X_{0}$. Utilizing Lemma 12, $W_{n}$ also has density bounded by $\ell<\infty$. Thus, we obtain that for some $\ell<\infty$ (not depending on $n$)

[TABLE]

From the definition of $q_{V_{n};m/\sqrt[\alpha]{n}}(x),q_{W_{n};m/\sqrt[\alpha]{n}}(x)$ in Definition 6, the densities of $q_{V_{n};m/\sqrt[\alpha]{n}}(x)$ and $q_{W_{n};m/\sqrt[\alpha]{n}}(x)$ are the average of the densities of $q_{V_{n}}(x)$ and $q_{W_{n}}(x)$ over the quantization interval. Therefore, they are also bounded from above by $\ell$:

[TABLE]

• •

There exists $v<\infty$ and $\gamma>0$ such that $\mathbb{E}\left[{\left|\widetilde{V}_{n;m/\sqrt[\alpha]{n}}\right|^{\gamma}}\right],\mathbb{E}\left[{\left|\widetilde{W}_{n;m/\sqrt[\alpha]{n}}\right|^{\gamma}}\right]\leq v$ for all $m,n$ satisfying $m/\sqrt[\alpha]{n}\geq 1$:

Utilizing Lemma 7 and the fact that $m/\sqrt[\alpha]{n}\geq 1$, we have that

[TABLE]

A similar equation holds for $\mathbb{E}\left[{\left|\widetilde{W}_{n;m/\sqrt[\alpha]{n}}\right|^{\gamma}}\right]$. Therefore, we will be done if we can find $v^{\prime}<\infty$ and $\gamma>0$ such that for all $m,n$ satisfying $m/\sqrt[\alpha]{n}\geq 1$, we have

[TABLE]

To this end, we can write

[TABLE]

Since $r_{n}<\frac{\sqrt[\alpha]{n}}{m}\leq 1$, we obtain

[TABLE]

Note that [26, Property 1.2.16]

[TABLE]

Therefore, the assertion is proved for $V_{n}$. Following a similar argument for $W_{n}$, we obtain

[TABLE]

Expected value of $|X_{0}|^{\gamma}$ is finite for any $\gamma<\alpha$. Thus, we only need to find some $\gamma<\alpha$ such that $\mathbb{E}\left[{|Y_{1}^{(n)}|^{\gamma}}\right]$ is finite. According to (25) in Lemma 4, we have

[TABLE]

Therefore,

[TABLE]

From (28) in Lemma 4, one can find some $\tilde{v}<\infty$ such that $\mathbb{E}\left[{|A_{n}|^{\gamma}}\right]<\tilde{v}$ for all $n$ provided that $\mathbb{E}\left[{|A|^{\gamma}}\right]<\infty$. Furthermore, $\sup_{n\in\mathbb{N}}n^{\frac{\gamma}{\alpha}}\left(1-\mathrm{e}^{-\frac{\lambda}{n}}\right)<\infty$ for any $\gamma<\alpha$. Therefore, we may choose any $0<\gamma<\alpha$.

∎

V-F Proof of Theorem 10

Proof of item 1: We can write that

[TABLE]

According to Theorem 6, there exists $c_{1},c_{2}>0$ such that

[TABLE]

where the second inequality is obtained from the fact that $\kappa_{Y}(n)=n\left(1-\exp\left[-\frac{\lambda_{Y}}{n}\left(1-\alpha_{Y}\right)\right]\right)$ tends to the constant $\lambda_{Y}(1-\alpha_{Y})$ as $n$ tends to infinity. Therefore,

[TABLE]

Hence, from the assumption of the theorem, we know that $\log m/\log n$ tends to $\infty$ as $n$ tends to infinity. As a result, it only suffices to prove that

[TABLE]

Equation (95) is immediate from Theorem 6 since for a discrete $X_{0}$ we have

[TABLE]

Furthermore, $\zeta_{X}(n)$ has a logarithmic growth from the second part of Theorem 6. To show (96), observe that from Theorem 6, we have

[TABLE]

which means that for any $\epsilon>0$, there exists $n_{1}$ such that

[TABLE]

In addition, note that $\zeta_{Y}(n)\geq c_{1}\log n$; therefore $\zeta_{Y}(n)+\log m$ converges to infinity as $n$ converges to infinity. Thus, for any $\epsilon>0$ there exists $n_{2}$ such that

[TABLE]

Hence, by taking $n=\max\{n_{1},n_{2}\}$, we have that

[TABLE]

Since $c_{1}\log n\leq\zeta_{Y}(n)\leq c_{2}\log n$, and $\log m/\log n$ tends to infinity if $m>m^{\prime}(n)$, we have

[TABLE]

Likewise,

[TABLE]

Therefore,

[TABLE]

Hence, (96) is proved.

Proof of item 2: We can write that

[TABLE]

It suffices to prove that

[TABLE]

Proof of (98): Note that from Theorem 6, we obtain that for any $\epsilon>0$, there exists $n_{1}$ such that

[TABLE]

In addition, $1/(n\log m)\leq\epsilon$ for any $m\geq 2$ and $n\geq\epsilon^{-1}$. Therefore, for any $\epsilon>0$ we have

[TABLE]

where $n_{2}=\max(n_{1},\epsilon^{-1})$. Utilizing the fact that from Theorem 6, $\zeta_{X}(n)\leq c_{2}\log n$, we have

[TABLE]

As a result,

[TABLE]

The value of $\kappa_{X}(n)$ given in Theorem 6 tends to a constant as $n$ tends to infinity. This completes the proof for (98).

Proof of (99): From the theorem’s assumption, the ratio of $\log m^{\prime}(n)$ and $\zeta_{Y}(n)$ goes to infinity. We know from Theorem 6 that $\zeta_{Y}(n)$ is non-decreasing. As a result,

[TABLE]

Utilizing Theorem 6 and (100), we obtain that for any $\epsilon>0$, there exists $n_{1}$ such that

[TABLE]

and

[TABLE]

As a result,

[TABLE]

We have that $0\leq\zeta_{Y}(n)/\log m\leq\zeta_{Y}(n)/\log m^{\prime}(n)$ for any $m>m^{\prime}(n)$. Furthermore, $\zeta_{Y}(n)/\log m^{\prime}(n)$ vanishes as $n$ tends to infinity according to the theorem’s assumption. Thus,

[TABLE]

Hence, the statement is proved.

Proof of item 3: We can write

[TABLE]

In order to prove the theorem, it suffices to show that

[TABLE]

[TABLE]

[TABLE]

From Theorem 6 and Theorem 7, the limits of (101) and (102) are proved, respectively.

To prove (103), observe that from Theorem 7

[TABLE]

where the first expression converges to $\mathrm{h}\left({Y_{0}}\right)$ according to Theorem 7. The second one tends to infinity because of the assumption of the theorem.

Finally, (104) is the direct consequence of Lemma 13.

Hence, the theorem is proved. ∎

V-G Proof of Theorem 11

Case 1: Stable

Proof of (20): According to Theorem 7, observe that for $i=1,2$

[TABLE]

Therefore, we can write

[TABLE]

Note that

[TABLE]

Since the limit of the second fraction depends on how $m$ changes as $n$ tends to $\infty$, it does not have a limit. However, due to $\alpha_{1}<\alpha_{2}$ we can write

[TABLE]

As a result, the statement is proved.

Proof of (21): We can write

[TABLE]

From Theorem 7 we obtain that there exists $n_{0}$ such that for $n>n_{0}$ we have

[TABLE]

where $C:=\mathrm{h}\left({X_{0}^{(\alpha_{1})}}\right)-\mathrm{h}\left({X_{0}^{(\alpha_{2})}}\right)-1$. Therefore, for $n>n_{0}$, we have that

[TABLE]

Hence, $\mathcal{H}_{m,n}(X_{1})-\mathcal{H}_{m,n}(X_{2})\to+\infty$.

Case 2: Impulsive Poisson

Proof of (22): Observe that for $i=1,2$ we have

[TABLE]

Hence, from Theorem 8, we can write

[TABLE]

Note that

[TABLE]

Hence, we obtain that

[TABLE]

Proof of (23): From Theorem 8 and the fact that

[TABLE]

we can write for $i=1,2$ that

[TABLE]

Then, we can write that

[TABLE]

Therefore, the first two parts are converging to a constant. For the last one, note that there exists $n_{0}$ such that for $n>n_{0}$ we have that while the last one converges to $-\infty$. Therefore, the theorem is proved. ∎

V-H Proof of Lemma 3

From Definition 9 and Theorem 3 we obtain that for any function $\varphi(t)\in\mathcal{S}(\mathbb{R})$

[TABLE]

Hence, from Definition 12, the assertion is proved. ∎

V-I Proof of Lemma 4

Proof of (25): First, we find the characteristic function of $Y_{n}$ in terms of the characteristic function of $A$. From the definition of $Y_{n}$ in (24), we have that

[TABLE]

where $\phi_{1,n}$ defined in Definition 13. Due to the definition of white noise in Definition 9, we can write

[TABLE]

Because of the definition of Poisson process in Definition 12, we have that

[TABLE]

where $\widehat{p}_{A}(\omega)$ is the characteristic function of $A$. Hence, we conclude that

[TABLE]

where

[TABLE]

Therefore, by taking the inverse Fourier transform of (105), (25) is proved.

Proof of (26): It is achieved by taking the inverse Fourier transform from (106).

Proof of (27): We can write

[TABLE]

Hence, we can write:

[TABLE]

Since $p_{A}$ is a probability density, $\overbrace{p_{A}*\cdots*p_{A}}^{k}$ is also a probability density; hence,

[TABLE]

Therefore,

[TABLE]

which vanishes as $n$ tends to $\infty$.

Proof of (28): Let the sequence of independent and identically distributed random variables $\left\{A^{(i)}\right\}_{i=1}^{\infty}$ have pdf $p_{A}$, and let the discrete random variable $B$ be independent to $\left\{A^{(i)}\right\}_{i=1}^{\infty}$ and have the following pmf:

[TABLE]

From (26), we obtain that

[TABLE]

Now, to find $\mathbb{E}\left[{\left|A_{n}\right|^{\alpha}}\right]$, we can write

[TABLE]

provided that the limit exists. From

[TABLE]

we have to prove that

[TABLE]

In order to prove (107), it suffices to show that

[TABLE]

where $c=2^{\alpha}(1+\mathbb{E}\left[{|A|^{\alpha}}\right])$ is a constant. This is because

[TABLE]

vanishes as $n$ tends to infinity. Therefore, it only remains to prove (108). We can bound \big{|}A^{(1)}+\cdots+A^{(k)}\big{|} as follows:

[TABLE]

By finding the expected value of the $\alpha$ power of both sides, we have

[TABLE]

where (109) is true because $\max\{1,|x|\}^{\alpha}=\max\{1,|x|^{\alpha}\}$ for all $x\in\mathbb{R}$, (110) is true because $A^{(1)},\cdots,A^{(n)}$ are independent, and (111) is true because $A^{(1)},\cdots,A^{(n)}$ are identically distributed. In order to find an upper bound for (111), we can write

[TABLE]

Hence, (108), and the theorem are proved. ∎

V-J Proof of Lemma 5

Proof of (29), (30), and (31): If we prove that $X_{0}$ is a stable random variable, then (29)-(31) are proved. [26] In order to do so, we find the characteristic function of $X_{0}$:

[TABLE]

where, $f(\omega)$ is a valid Lévy exponent, defined in (3), and (112) is true because of the definition of stable white noise in Definition 11. Hence, from Definition 10, we obtain that $X_{0}$ is a stable random variable.

Proof of (32): Since the functions $x\mapsto p_{X_{0}}(x)$ and $p\mapsto p|\log(1/p)|$ are continuous, the function $x\mapsto p_{X_{0}}(x)\left|\log(1/p_{X_{0}}(x))\right|$ is also continuous. Thus, for any arbitrary $x_{0}>0$, we have

[TABLE]

Therefore, in order to prove (32), we need to prove the boundedness of the tail of the integral. For sufficiently large $x_{0}$, we have

[TABLE]

for some positive constant $c$ depending on parameters $(\alpha,\beta,\sigma,\mu)$ [26]. Since $p\mapsto p|\log({1}/{p})|$ is increasing for $p\in[0,1/\mathrm{e}]$, it suffices to show that for sufficiently large $x_{0}$, the following integral is bounded:

[TABLE]

By changing variable $y={x}/{c^{{1}/(\alpha+1)}}$, it suffices to show that

[TABLE]

where $y_{0}={x_{0}}/{c^{{1}/(\alpha+1)}}$, which holds.

Proof of (33): To show that $\mathrm{H}\left({{\left[{X_{0}}\right]}_{1}}\right)<\infty$, it suffices to look at the tail of the probability sequence of ${\left[{X_{0}}\right]}_{1}$. From (113), for sufficiently large $x_{0}$, we have that for all $|m|>x_{0}$

[TABLE]

Hence, it suffices to show that

[TABLE]

due that ${a}/{x^{\alpha+1}}\log({x^{\alpha+1}}/{a})$ is a decreasing function for sufficiently large $x$, we obtain that

[TABLE]

where the last inequality is true because of (114). ∎

V-K Proof of Lemma 6

Using Definition 11, the characteristic function of $X_{1}^{(n)}$ is as follows:

[TABLE]

From the definition of $X_{0}$, we can write

[TABLE]

Thus, we obtain that

[TABLE]

where

[TABLE]

Therefore, $X_{1}^{(n)}$ can be written with respect to $X_{0}$ as follows:

[TABLE]

where $b_{n}=c_{n}\sqrt[\alpha]{n}$. ∎

V-L Proof of Lemma 7

Proof of (34): From the definition of $q_{X;m}$ and $q_{Y;m}$, defined in Definition 6, it follows that they are constant in intervals $\left[(i-{1}/{2})/{m},(i+{1}/{2})/{m}\right]$ for all $i\in\mathbb{Z}$. Therefore,

[TABLE]

where (117) is from the definition of $q$. Hence by summation over $i\in\mathbb{Z}$, we have

[TABLE]

Thus, (34) is proved.

Proof of (35) and (36): We claim that, it suffices to show that

[TABLE]

Because if so, we can write

[TABLE]

and as a result

[TABLE]

Therefore, it suffices to show that

[TABLE]

[TABLE]

Proof of (119): By conditioning whether $|X|>1/\sqrt{m}$ or not, we can write

[TABLE]

where (121) is true since

[TABLE]

Note that if $\mathrm{Pr}\left\{{|X|>\tfrac{1}{\sqrt{m}}}\right\}=0$, the conditional expected value \mathbb{E}\big{[}\big{(}|X|+\tfrac{1}{m}\big{)}^{\alpha}\big{|}|X|>\tfrac{1}{\sqrt{m}}\big{]} is not well-defined, but in this case we take the product \mathrm{Pr}\left\{{|X|>\tfrac{1}{\sqrt{m}}}\right\}\mathbb{E}\big{[}\big{(}|X|+\tfrac{1}{m}\big{)}^{\alpha}\big{|}|X|>\tfrac{1}{\sqrt{m}}\big{]} to be zero, without any need for specifying an exact value for \mathbb{E}\big{[}\big{(}|X|+\tfrac{1}{m}\big{)}^{\alpha}\big{|}|X|>\tfrac{1}{\sqrt{m}}\big{]}.

From (121), it only remains to prove that

[TABLE]

In order to do that, we can write

[TABLE]

Since $|X|>1/\sqrt{m}$, we can write

[TABLE]

where the last inequality is true because $\log(1+x)\leq x$. As a result

[TABLE]

Proof of (120): Similar to the previous proof, by conditioning whether $|X|>1/\sqrt{m}$ or not, we can write

[TABLE]

If $\mathrm{Pr}\left\{{|X|>1/\sqrt{m}}\right\}=0$, (120) is clearly correct. Thus, assume that $\mathrm{Pr}\left\{{|X|>1/\sqrt{m}}\right\}>0$, meaning that \mathbb{E}\left[{||X|-{1}/{m}|^{\alpha}\big{|}|X|>{1}/{\sqrt{m}}}\right] is well-defined. We need to show that

[TABLE]

Observe that when $|x|>1/\sqrt{m}$ and $m\geq 4$, we can write

[TABLE]

where the last inequality is true because $\log(1-x)\geq-2x$ for $0\leq x\leq 1/2$. As a result,

[TABLE]

Now, we need to show that

[TABLE]

Without loss of generality, we may assume that $\mathrm{Pr}\left\{{|X|>1/\sqrt{m}}\right\}<1$. Now, we have

[TABLE]

Since

[TABLE]

we obtain that

[TABLE]

Therefore, (123) is obtained from (124). Hence, (120) is proved.

Proof of (118): Note that random variable $\widetilde{X}_{m}$ has the same distribution as the following random variable:

[TABLE]

where $U_{m}$ and $X$ are independent, and

[TABLE]

Thus,

[TABLE]

where the last inequality is true because $\left|{\left[{X}\right]}_{m}-X\right|$ and $|U_{m}|$ are always less than ${1}/{2m}$. Therefore, the lemma is proved. ∎

V-M Proof of Lemma 8

From the definition of entropy, we know that the entropy of a random variable does not depend on the value of the random variable, rather it only depends on the distribution of the random variable. Therefore, if one finds a correspondence between the values of ${\left[{aX}\right]}_{m/a}$ and the values of ${\left[{X}\right]}_{m}$, while they have the same probability, then the entropy of them will be the same. Thus, we define the following correspondence between the values of ${\left[{aX}\right]}_{m/a}$, which are from the set $\left\{k\frac{a}{m}\left|k\in\mathbb{Z}\right.\right\}$, and the values of ${\left[{X}\right]}_{m}$, which are from the set $\left\{k\frac{1}{m}\left|k\in\mathbb{Z}\right.\right\}$.

[TABLE]

Now, we show that the correspondent values have the same probability.

[TABLE]

Therefore, the lemma is proved. ∎

V-N Proof of Lemma 9

Observe that for any given $m,c_{m}\in\mathbb{R}$, there exist unique $k_{m}\in\mathbb{Z}$, and $d_{m}\in\mathbb{R}$ such that

[TABLE]

Therefore, because of the definition of amplitude quantization in Definition 6, we can write

[TABLE]

Since entropy is invariant with respect to constant shift, we have

[TABLE]

As a result, we only need to prove the theorem for $\{d_{m}\}$, instead of $\{c_{m}\}$. Take the pdf $q_{m}(x)$ as follows:

[TABLE]

Therefore, from Lemma 1, we obtain that

[TABLE]

where the last equation is true because of (125). So, if we take $\widetilde{X}_{m}$ a continuous random variable with pdf $q_{m}$, we only need to prove

[TABLE]

In order to do this, we write $\left|\mathrm{h}\left({\widetilde{A}_{m}}\right)-\mathrm{h}\left({X}\right)\right|$ as follows

[TABLE]

where $l>0$ is arbitrary. Thus, it suffices to prove that

[TABLE]

[TABLE]

[TABLE]

hold for all $l>0$ and uniformly on $m$.

Proof of (126): From (125), we obtain that since $p(x)$ is a piecewise continuous, the mean value theorem yields that there exists $x^{*}_{m}\in\left[(i-\tfrac{1}{2})/m-d_{m},(i-\tfrac{1}{2})/m-d_{m}\right)$, such that $q_{m}(x)=p(x^{*}_{m})$. Since $p(x)$ is piecewise continuous, and $[-l,l]$ is a compact set, $p(x)$ is uniformly continuous over $[-l,l]$. Therefore, for any $\epsilon^{\prime}$, there exists $M\in\mathbb{R}$ such that for all $x\in[-l,l]$, we have that

[TABLE]

Furthermore, since the function $x\mapsto x\log x$ is continuous and $p(x)$ is uniformly continuous, we have that for all $x\in[-l,l]$

[TABLE]

uniformly on $x$. Thus (126) is proved.

Proof of (127): We can write

[TABLE]

By assuming the lemma, we know

[TABLE]

Hence, (129) implies (127).

Proof of (128): It suffices to show that

[TABLE]

where $o_{l}(1)$ means that $\lim_{l\to\infty}{o_{l}(1)}=0$, and

[TABLE]

By changing the variable $y=Lx$, we can write

[TABLE]

Because $p(x)<L$ almost everywhere, we have that:

[TABLE]

So we obtain that

[TABLE]

where $x=y/L$. Therefore, we can write that

[TABLE]

Thus, from (130) we conclude that the lower bound vanishes as $l$ tends to $\infty$ uniformly on $m$. Now, we are going to show that (132) leads that the upper bound vanishes as $l$ tends to $\infty$ uniformly on $m$. In order to prove this, note that $\mathrm{Pr}\left\{{|X|>l-1}\right\}$ vanishes as $l$ tends to infinity uniformly on $m$. Furthermore,

[TABLE]

Thus, in order to prove (128), it only remains to prove (131) and (132). The proof of (131) exists in [1] in the proof of Theorem 1. Thus, we only need to prove (132). Similar to the proof of Theorem 1 in [1], it can be shown that

[TABLE]

where $P_{m}[i]:=\mathrm{Pr}\left\{{{\left[{X+d_{m}}\right]}_{1}=i}\right\}$. This implies

[TABLE]

where $\mathrm{H}\left({X|Y=y}\right)$ is defined in [24, p. 29]. Hence, we can write

[TABLE]

Because of the definition of the quantization, we can write

[TABLE]

where $E_{m}$ is a random variable taking values from $\{0,1\}$. Thus, ${\left[{X+d_{m}}\right]}_{1}$ is a function of ${\left[{X}\right]}_{1}$ and $E_{m}$; as a result

[TABLE]

From the definition of \mathrm{H}\left({{\left[{X}\right]}_{1}\big{|}|{\left[{X+d_{m}}\right]}_{1}|>l}\right), we can write that

[TABLE]

where

[TABLE]

Therefore, (133) can be simplified as follows:

[TABLE]

The first term, (134), vanishes as $l$ tends to $\infty$ because

[TABLE]

It can be achieved that (135) and (136) vanish as $l$ tends to $\infty$, uniformly on $m$, but we do not write the details here. Thus, (132) is proved and the proof of lemma is complete. ∎

V-O Proof of Lemma 11

According to Lemma 10, we can define random variable $U$ with support $\{\mathrm{dc},\mathrm{cd},\mathrm{cc}\}$ and pmf $p_{U}(u)$ such that

[TABLE]

and

[TABLE]

where

[TABLE]

Thus, we have that

[TABLE]

Hence, the lemma is proved. ∎

V-P Proof of Lemma 12

From the Fubini’s theorem we obtain that

[TABLE]

where $F_{X}(x):=\mathrm{Pr}\left\{{X\leq x}\right\}$ is the cdf of $X$. In order to prove the first part of the lemma, we only need to prove that

[TABLE]

The above integral can be written as the following limit:

[TABLE]

Note that, from Fubini’s theorem, for every finite $\ell$, we have that

[TABLE]

Hence, to prove (137), it is sufficient to show that

[TABLE]

In order to do so, note that there exists $\ell_{X}$, $\ell_{Y}$ such that

[TABLE]

Thus, we can write

[TABLE]

Therefore, the the lemma is proved. ∎

V-Q Proof of Lemma 13

Note that by the entropy power inequality (EPI) [24, Theorem 17.7.3], we have that for $n$ iid scalar continuous random variables $X_{1},\cdots,X_{n}$

[TABLE]

Therefore, from Lemma 2 we have that

[TABLE]

where $X_{1}^{(n)}$ was defined in Definition 13. According to Theorem 6, $\zeta(n)=\mathrm{h}\left({X_{1}^{(n)}}\right)$ satisfies (17). This will complete the proof since based on Theorem 5, for any $\zeta^{\prime}(n)$ satisfying (17), we have that

[TABLE]

Hence,

[TABLE]

Thus, the lemma is proved. ∎

VI Conclusion

In this paper, a definition of quantization entropy for random processes based on quantization in the time and amplitude domains was given. The criterion was applied to a wide class of white noise processes, including stable and impulsive Poisson innovation processes. It was shown that the stable has a higher growth rate of entropy compared to the impulsive Poisson process. In our study, we assumed that the amplitude quantization steps $1/m$ is shrinking sufficiently fast with respect to time quantization steps $1/n$, i.e., $m$ is larger than $m(n)$ for some function $m:\mathbb{N}\to\mathbb{N}$. As a future work, it would be interesting to look at cases where $m$ is restricted grow slowly with $n$. Characterization of the entropy for other stochastic processes is also left as a future work.