Entropy of Independent Experiments, Revisited

TL;DR

This paper analyzes the convergence rate of Renyi entropy to Shannon entropy for independent, non-i.i.d. sources, identifying worst-case distributions and improving bounds in high-entropy regimes.

Contribution

It characterizes sources with slowest entropy convergence, improves bounds in high-entropy regimes, and applies non-convex programming to derive extremal inequalities.

Findings

Identified distributions with slowest convergence of entropy.

Established faster convergence bounds in high-entropy regimes.

Provided tools for better security quantification of random number generators.

Abstract

The weak law of large numbers implies that, under mild assumptions on the source, the Renyi entropy per produced symbol converges (in probability) towards the Shannon entropy rate. This paper quantifies the speed of this convergence for sources with independent (but not iid) outputs, generalizing and improving the result of Holenstein and Renner (IEEE Trans. Inform. Theory, 2011). (a) we characterize sources with \emph{slowest convergence} (for given entropy): their outputs are mixtures of a uniform distribution and a unit mass. (b) based on the above characterization, we establish faster convergences in \emph{high-entropy} regimes. We discuss how these improved bounds may be used to better quantify security of outputs of random number generators. In turn, the characterization of "worst" distributions can be used to derive sharp "extremal" inequalities between Renyi and Shannon entropy. The main technique is \emph{non-convex programming}, used to characterize distributions of possibly large exponential moments under certain entropy.

Tables

Table 1. Table 1 : Summary of our bounds and comparison with related works. The alphabet size is | 𝒳 | = N 𝒳 𝑁 |\mathcal{X}|=N , the number of samples n 𝑛 n is such that ( 2 ) holds with accuracy δ 𝛿 \delta and error probability ϵ italic-ϵ \epsilon , and defficiencies are are bounded by Δ Δ \Delta . Note that our bounds for the low entropy setting Δ = log ⁡ N Δ 𝑁 \Delta=\log N reduce to [ HR11 ] .

author/reference	number of samples $n$	regime	technique
[HR11]	${\log }^{2}N\cdot {\delta }^{-2}\log \frac{1}{ϵ}$		bounds on exponential moments
this paper, Corollary 1 (a)	$\log \left(\Delta N\right)\cdot \max ({\delta }^{-1},\Delta {\delta }^{-2})\log \frac{1}{ϵ}$	$\Delta >\frac{1}{N}$	optimized exponential moments subexpoential tails
this paper, Corollary 1 (b)	$\max (\Delta \cdot {\delta }^{-1},1)\log \frac{1}{ϵ}$

Table 2. Table 2 : Entropy defficiency as a function of bias.

bias	support	regime	entropy deficiency
$\gamma$	$N$	$\gamma N=\omega \left(1\right)$	$\Delta =\Theta \left(\gamma \log \gamma N\right)$
		$\gamma N=\Theta \left(1\right)$	$\Delta =\Theta \left(\gamma \right)$
		$\gamma N=O\left(1\right)$	$\Delta =\Theta \left({\gamma }^{2}N\right)$

Equations

$$\frac{1}{n}\log \frac{1}{P_{X^n}(x)}=\frac{1}{n}H(X^n)+o(1)\text{w.p. }1-o(1)$$

$$\underset{x_1,\dots ,x_n\leftarrow X_1,\dots ,X_n}{\mathrm{Pr}}\left[\left|\sum _{i=1}^n\log \frac{1}{P_{X_i}(x_i)}-\sum _{i=1}^{n}H(X_i)\right|>n\delta \right]⩽ϵ$$

$$\underset{X}{\max }{\mathbb{E}}_{x\sim X}\exp \left(t\log \frac{1}{P_X(x)}\right)⩽?$$

$$\begin{array}{rlr}\underset{X}{\mathrm{maximize}}& & {\mathbb{E}}_{x\sim X}\exp \left(tH(X)-t\log \frac{1}{P_X(x)}\right)\\ s.t.& & H(X)=k.\end{array}$$

$$\displaystyle P_{X}(x)=\left\{\begin{array}[]{r}\theta,\ x=x_{0}\\ \frac{1-\theta}{|\mathcal{X}|-1},\ x\not=x_{0}\\ \end{array}\right.$$

$$L=\sum _i{p}_i\exp \left(tk-t\log \frac{1}{p_i}\right)+\lambda \left(H(p)-k\right)$$

$$\displaystyle\epsilon=\left\{\begin{array}[]{rl}\exp\left(-\frac{t^{2}}{2n\Delta\log(N\Delta)}\right),&\quad t<n\Delta\\ \exp\left(-\frac{t}{2\log(N\Delta)}\right),&\quad t>n\Delta\end{array}\right.$$

$$\displaystyle\epsilon=\left\{\begin{array}[]{rl}\exp\left(-\frac{t^{2}}{2n\Delta}\right),&\quad t<n\Delta\\ \exp\left(-\frac{t}{2}\right),&\quad t>n\Delta\end{array}\right.$$

$$\mathbb{E}\exp \left(t(X-\mu )\right)⩽\exp \left(\frac{{\sigma }^2{t}^2}{2}\right).$$

$$\displaystyle\Pr[X>\mu+t]\leqslant\left\{\begin{array}[]{rl}\exp\left(-\frac{t^{2}}{2\sigma^{2}}\right),&0\leqslant t\leqslant\frac{\sigma^{2}}{b}\\ \exp\left(-\frac{t}{b}\right),&\frac{\sigma^{2}}{b}<t\end{array}\right.$$

$$\displaystyle\begin{aligned} \underset{p\in D}{\mathrm{maximize}}&&f(p)\\ \mathrm{s.t.}&&\begin{array}[]{l}h_{i}=0,\quad i\in I\\ g_{j}\geqslant 0,\quad j\in J.\end{array}\end{aligned}$$

$$L=f+\sum _i{\lambda }_i\cdot h_i+\sum _j\cdot {\mu }_j{g}_j$$

$$\frac{\partial L}{\partial p}(p^{\ast })=0.$$

$$d^T\cdot \frac{{\partial }^{2}L}{\partial p^2}(p^{\ast })\cdot d⩾0$$

$$\sum _{i=1}^j{u}_i^{\prime }⩾\sum _{i=1}^j{v}_i^{\prime }\text{ for }j=1,\dots ,d$$

$$\forall x:P_X(x)\in \{\alpha ,\beta \},\#\{x:P_X(x)=\alpha \}=1$$

$$\displaystyle\begin{aligned} \underset{p}{\mathrm{maximize}}&&\sum_{i}p_{i}\mathrm{e}^{tH(p)-t\log\frac{1}{p_{i}}}\\ \mathrm{s.t.}&&\left.\begin{array}[]{r}H(p)=k\end{array}\right.\end{aligned}$$

$$\sum _i{q}_i{\mathrm{e}}^{tH(q)-t\log \frac{1}{q_i}}>{\mathrm{e}}^{t{\sum }_i{q}_i\left(H(q)-\log \frac{1}{q_i}\right)}={\mathrm{e}}^{tH(q)-tH(q)}=1.$$

$$\delta \log \frac{1}{\delta }+(1-\delta )\log \frac{N-1}{1-\delta }=k$$

$$\displaystyle\begin{aligned} \underset{p}{\mathrm{maximize}}&&\sum_{i}p_{i}^{1+t}\\ \mathrm{s.t.}&&\left.\begin{array}[]{r}H(p)=k\end{array}\right.\end{aligned}$$

$$\forall i:p_i^{\ast }\in \{0,\alpha ,\beta \},\#\{i:p_i^{\ast }=\alpha \}=1$$

$$\displaystyle\begin{aligned} \underset{p}{\mathrm{maximize}}&&\sum_{i\in I}p_{i}^{1+t}\\ \mathrm{s.t.}&&\left\{\begin{array}[]{r}H(p)=k\\ \sum_{i\in I}p_{i}=1\\ \forall i\in I:\quad p_{i}>0\end{array}\right.\end{aligned}$$

$$L(p,{\lambda }_1,{\lambda }_2)=\sum _{i\in I}{p}_i^{1+t}-{\lambda }_1\left(\sum _{i\in I}{p}_i\log \frac{1}{p_i}-k\right)-{\lambda }_2\left(\sum _{i\in I}{p}_i-1\right).$$

$$\forall i\in I:0=\frac{\partial L}{\partial p_i}(p^{\ast })=(1+t)(p^{\ast }{)}_i^t-{\lambda }_1\left(\log \frac{1}{p_i^{\ast }}-1\right)-{\lambda }_2$$

$$\displaystyle\left.\begin{array}[]{r}N_{\alpha}\alpha+N_{\beta}\beta=1\\ N_{\alpha}\alpha\log\frac{1}{\alpha}+N_{\beta}\beta\log\frac{1}{\beta}=k\end{array}\right.$$

$$\displaystyle\mathrm{det}\left[\frac{\partial g}{\partial(\alpha,\beta)}\right]=\mathrm{det}\left[\begin{array}[]{rl}N_{\alpha}&\ N_{\alpha}\left(\log\frac{1}{\alpha}-1\right)\\ N_{\beta}&\ N_{\beta}\left(\log\frac{1}{\beta}-1\right)\end{array}\right]=N_{\alpha}N_{\beta}\left(\log\frac{1}{\beta}-\log\frac{1}{\alpha}\right).$$

$$\displaystyle\frac{\partial^{2}L}{\partial p_{i}p_{j}}=\left\{\begin{array}[]{rl}0,&i\not=j\\ t(1+t)p_{i}^{t-1}+\lambda_{1}p_{i}^{-1},&i=j.\end{array}\right.$$

$$\displaystyle\begin{array}[]{r}t(1+t)\alpha^{t-1}+\lambda_{1}\alpha^{-1}=0\\ (1+t)\alpha^{t}-\lambda_{1}\left(\log\frac{1}{\alpha}-1\right)-\lambda_{2}=0\\ (1+t)\beta^{t}-\lambda_{1}\left(\log\frac{1}{\beta}-1\right)-\lambda_{2}=0\end{array}$$

$$t{\alpha }^t-\frac{{\alpha }^t-{\beta }^t}{\log \alpha -\log \beta }=0.$$

Full text

This paper quantifies the speed of this convergence for sources with independent (but not iid) outputs, generalizing and improving the result of Holenstein and Renner (IEEE Trans. Inform. Theory, 2011).

1. (a)

we characterize sources with slowest convergence (for given entropy): their outputs are mixtures of a uniform distribution and a unit mass. 2. (b)

based on the above characterization, we establish faster convergences in high-entropy regimes.

We discuss how these improved bounds may be used to better quantify security of outputs of random number generators.

In turn, the characterization of ”worst” distributions can be used to derive sharp ”extremal” inequalities between Renyi and Shannon entropy.

The main technique is non-convex programming, used to characterize distributions of possibly large exponential moments under certain entropy.

symptotic Equipartition Property, Smooth Entropy, Exponential Moments Method

1 Introduction

It is well known that under mild assumptions on the source (independent and identical outputs [RW04], independent but not identical outputs [HR11], ergodicity [STTV07], strong converse property [Kog13]) the rate of min-entropy (in fact, Renyi entropy of any positive order) converges in probability111For the convergence in probability we understand the entropy as conditioned over some set of probability $1-o(1)$. This is strictly related to smooth entropy [RW04].. towards the Shannon entropy rate. More precisely, for the source producing outputs $X_{1},X_{2},\ldots$ and $x\leftarrow X^{n}\overset{def}{=}X_{1},\ldots,X_{n}$, under these assumptions for $n\to\infty$ we have

[TABLE]

which can be seen a demonstration of the weak law of large numbers222By the definition of Shannon entropy we have $\mathbb{E}\log\frac{1}{P_{X^{n}}(\cdot)}=H(X^{n})$..

In information theory results of this sort are often refereed to as generalizations of the Asymptotic Equipartition Property, because they establish that with overwhelming probability the sequences produced by the source are (roughly) equally likely.

Under general assumptions, there is basically not much more to say about (1). This paper is concerned with quantitative bounds, which are possible when the source produces independent outputs. Thus, we are interested in inequalities

[TABLE]

where $X_{i}$ are independent random variables with finitely many outcomes333Our results are valid when $X_{i}$ have different alhapbets, however for the clarity of the presenation we later assume that they all are over some fixed $\mathcal{X}$.. Bounds of this sort find applications in cryptography, quantifying the the conversion between Shannon entropy (more convenient to quantify) and the min-entropy (required for security), for a series of experiments. One example are theoretical constructions of pseudorandom generators [HILL99, Hol06], which use a variant of (2). Another important application is a justification of the entropy evaulation methodology for random number generators [KS11, TBKM16], where best available tests quantify Shannon entropy [Mau92, CN99].

Good bounds of the form (2) are obtained by techniques from large deviations theory, applied to the random variables $Z_{i}=\log\frac{1}{P_{X_{i}}(\cdot)}$ called surprises of $X_{i}$. Note that $Z_{i}$ are unbounded, hence standard inequalities like Chernoff-Hoeffding bounds don’t apply. The solution444There are other approaches, for example ignoring large surprises or using the concept of typical sets, but they lead to worse bounds as discussed in [HR11]. is to work directly with moment generating functions of each $Z_{i}$. If we know upper bounds on $\mathbb{E}\exp(tZ_{i})$, where $t$ is a parameter, then (2) follows by the Markov inequality $\Pr[\sum_{i}Z_{i}\geqslant n\delta]\leqslant\prod_{i}\exp\left(tZ_{i}\right)\cdot\exp(-tn\delta)$, optimized over $t$. In bounds (2) we may also want to capture some information about the Shannon entropy of the source. Technically, the problem then reduces to

(Problem) For any alphabet $\mathcal{X}$, find best possible bounds on the exponential moments of the surprise of a distribution $X$ over $\mathcal{X}$

$\displaystyle\underset{X}{\textrm{max}}\ \mathbb{E}_{x\sim X}\exp\left(t\log\frac{1}{P_{X}(x)}\right)\leqslant\quad?$

assuming that $X$ has Shannon entropy $k$, for parameters $k$ and $t$.

We follow this approach and derive best possible bounds for this technique.

1.1 Related works and our results

1.1.1 Related works

Best bounds of the form (2) so far were due to Holenstein and Renner [HR11]. Their argument uses calculus to derive bounds on the exponential moments of the surprise of $X$. This leads to $\epsilon=\exp\left(-\Omega(1)\cdot\frac{n\delta^{2}}{\log^{2}|\mathcal{X}|}\right)$. However the matching (up to a constnat in the exponent) lower bound $\epsilon=\exp\left(-O(1)\cdot\frac{n\delta^{2}}{\log^{2}|\mathcal{X}|}\right)$ is known only for low or moderate entropy rates555Note that in [HR11] the bounds are shown to be optimal only for moderate entropy, namely of about $\frac{1}{2}\log|\mathcal{X}|$ per sample (cf the proof of Theorem 3 in [HR11]..

1.1.2 Our results and techniques in a nutshell

We provide explicit bounds in terms of entropies of $X_{i}$, instead of the alphabet as in [HR11]. Roughly speaking, we replace the factor logarithmic in the alphabet by the entropy efficiency.

In particular, we obtain significant improvements in high-entropy regimes where the deficiency $\Delta_{i}=H_{0}(X_{i})-H(X_{i})$ is relatively small, for example a fraction of the length (here $H_{0}(\cdot)$ is the logarithm of the support of $X_{i}$). As a consequence, (1) converges faster when the distributions $X_{i}$ have high entropy; alternatively, we get better accuracy $\delta$ with fewer samples. We summarize our bounds in Table 1.

The bounds are relevant to random number generation, where we improve the known relation between min-entropy (which is the best notion to be used in vulnerability analysis) and Shannon entropy (much easier to estimate in practice [KS11]). We discuss this application in the next paragraph.

To quantify the convergence in terms of entropies of $X_{i}$, we use non-convex optimization to find best bounds on exponential moments of $X_{i}$ under given entropy. While the optimization is rather non-trivial, the extreme distribution has a simple shape: it combines a unit mass and a uniform distribution. This analysis not only guarantees that the bounds are best possible, but as a byproduct solves the related problem of Renyi entropy maximization under fixed Shannon entropy. This way we obtain extremal inequalities between these entropies.

1.1.3 Applications to random number generators

The motivation for studying high entropy regimes comes from true random number generators. Roughly speaking, they are devices which postprocess samples from a physical entropy source into a sequence of almost independent and unbiased bits.

1. (a)

output security: output bits are required to have very high Shannon entropy666 per bit (e.g. more than 0.997) and no dependencies [KS11]; for example, early Intel (hardware) generators were estimated to generate about 0.999 Shannon entropy per bit [jun1999intel].

To illustrate our bounds, suppose that outputs are generated in $8$ bits chunks. We have $|\mathcal{X}|=2^{8}$ states and the Shannon entropy deficiency is $\Delta=(1-0.997)\cdot 8=0.024$. The security (min-entropy) implied by bounds in Table 1 at the confidence $1-\epsilon$ where $\epsilon=2^{-60}$ is illustrated in Figure 1 below.

2. (b)

source health tests: our bounds may be also applied to quantify decreases in the source quality (standards [TBKM16, KS11] require such countermeasures to be implemented). Namely, our bounds provide the min-entropy rate in the ideal case (independent samples), which upperbounds the actual rate. In practical designs one prefers sources with small entropy deficiency [HHL13, PJO15, BS15], where our results offer more accurate estimates.

1.1.4 Applications to extremal inequalities for Renyi entropy

As a byproduct of our analysis, we obtain sharp bounds on Renyi entropy for given Shannon entropy, in one-shot experiments (as opposed to the previous application). The motivation comes from cryptographic tasks such as key derivation, which demand ”enough” Renyi entropy available. We ask what can be said about Renyi entropy of a distribution, if only its Shannon entropy is known. Using our techniques, the precise answer can be given for any $\alpha>0$. Below in Figure 2 we illustrate such a bound for Renyi entropy of order $\alpha=2$ (denoted by $H_{2}$). The Python script is included in Appendix 0.B.

1.1.5 Our results in detail

Optimal bounds on exponential moments of surprises

We compute the maximal value of the moment generating function of the surprise, when the distribution is over the finite space $\mathcal{X}$ and has a certain ammount of entropy $k$. Essentialy, we need to solve the following optimization task777This program is equivalent to what we announced in the introduction.

[TABLE]

over random variables $X$ taking values in $\mathcal{X}$, where $t$ is a parameter. Our main result characterizes the optimal solution to (3).

Theorem 1.1 (Sharp surprise exponential moments)

*For any $t\geqslant-1$, the optimal solution to (3) is given by *

[TABLE]

for some $\theta\in\left(\frac{1}{|\mathcal{X}|},1\right)$ and $x_{0}\in\mathcal{X}$.

Remark 1 (Intuitions)

The result essentially says that the optimal distribution is a combination of a unit mass and a uniform distribution. This is a consequence that the optimization program in (3) basically exhibits two different behaviors: concavity for small probability weights and convexity for larger probability weights. While this charecterization is simple the proof is not, as the standard convex/concave programming framework cannot be applied.

Remark 2 (Techniques)

To handle constrained optimization like (3), the standard approach is to skip the constraint, adding instead a corresponding penalty term to the objective (forming the so called Lagrangian). In our case we get

[TABLE]

for a weight $\lambda$, to be maximized over probability vectors $p$ (the dual problem).

By the elegant methods of majorization theory we show that the dual problem is solved by a distribution as in Equation 6. Basically this is because $L$ as a function of $p$ is convex when restricted to variables $p_{i}>c$ and concave when restricted to variables $p_{i}<c$, where $c$ is some constant. To maximize in the convex region the best choice is to have only one $i$ such that $p_{i}>c$. To maximize in the concave region the optimal choice is $p_{i}=p_{j}<c$ when $p_{i},p_{j}<c$ (see Figure 3 below).

Unfortunatelly, our $L$ fails to satisfy desired convexity/concavity properties and the solution to the dual problem is not guaranteed to be optimal for the original problem. To rule out this possibility, called the duality gap, we use second order conditions. These conditions are basically a variational analysis of how $L$ changes when a stationary point $p$ moves a little bit in a way consistent with the constraints (to make this precise, the constraints are linearized and $L$ is approximated up to second order terms). In our case we conclude that $p$ must be anyway as in (3), which completes the proof.

The argument is technically more complicated than sketched here, as we need to make sure that the optimal $p$ has no zero weights (by restricting to positive weights we avoid singularities in anaylysis). We illustrate our proof in Figure 4 below. The details are given in Section 3.0.1.

1.1.6 Optimal sub-exponential tails under entropy constaints

From Equation 6 we derive best possible bounds on the surprise exponential moments. There are two technical difficulties to handle. The first issue is that the distribution in (6) is given in terms of the bias $\gamma=\theta-\frac{1}{|\mathcal{X}|}$, whereas we need to parameterize it in terms of the entropy amount $k$. Resolving the equation $H(X)=k$ involves inverting non-elementary equations of the form $\theta\log\theta=c$, and can be done with the use of the Lambert-W function. The second issue is that plugging (6) into (3) does not lead to a clean formula, and requires some calculus to get clear bounds.

Corollary 1

For independent sequence $X_{1},X_{2},\ldots,X_{n}$ of random variables over $\mathcal{X}$ with entropy defficiency $\Delta_{i}=H_{0}(X_{i})-H(X_{i})$ at most $\Delta$, we have (2) with the following parameters (below $N=\log|\mathcal{X}|$)

1. (a)

for $\Delta=\omega(N^{-1})$ the tail is

[TABLE] 2. (b)

when $\Delta=O(N^{-1})$ the tail is

[TABLE]

The bounds in Table 1 follow by setting $t=n\delta$ and by combining the formulas for $t<n\Delta$ and $t>n\Delta$ into one, with the maximum function.

Remark 3

Examples To illustrate the bounds, consider the minimal number $n$ to have (2), for fixed accuracy $\delta$ and error probability $\epsilon)$-approximation

1. (a)

when $\Delta_{i}=O(1)$ (constant defficiency) we get $n=O(1)\log|\mathcal{X}|\delta^{-2}\log(1/\epsilon)$, saving a factor of $\Omega(\log|\mathcal{X}|)$ comparing to [HR11] 2. (b)

when $\Delta_{i}=O\left(\frac{1}{H_{0}(X_{i})}\right)$ then $n=O(1)\delta^{-2}\log(1/\epsilon)$, which doesn’t depend on the alphabet anymore.

The proof appears in Section 3.1.

1.2 Organization

We provide background definitions and auxiliary fact used in Section 2. The proofs are given in Section 3. We conclude our work in Section 4

2 Preliminaries

We say that the vector $p=(p_{i})_{i}$ is a probability vector if its entries are non-negative and add up to 1. The distribution of a radom variable $X$ is denoted by $P_{X}(x)=\Pr[X=x]$. The surprise of $X$ is a random variable $x\rightarrow\log\frac{1}{P_{X}(x)}$. The Shannon entropy is the expected surprise $\mathbb{E}_{x\sim X}\log\frac{1}{P_{X}(x)}=\sum_{x}P_{X}(x)\log\frac{1}{P_{X}(x)}$. The Renyi entropy of order $\alpha$ is defined as $-\frac{1}{\alpha-1}\log\sum_{x}P_{X}(x)^{\alpha}$.

2.1 Sub-Gaussian Random Variables

Below we remind basic facts from the theory of subgaussian and subexponential distributions (we refer to [Ver16] for a detailed threatment).

Definition 1 (Sub-gaussian tails)

A real-valued random variable $X$ with mean $\mu$ is sub-gaussian with parameter $\sigma^{2}$ if for all real $t$ we have

[TABLE]

Lemma 1 (Independent sub-gaussian random variables)

For independent $X_{i}$ each sub-gaussian with parameters $\sigma_{i}^{2}$, the sum $X=\sum_{i}X_{i}$ is sub-gaussian with parameter $\sum_{i}\sigma_{i}^{2}$.

2.2 Sub-Exponential Random Variables

Definition 2 (Sub-exponential random variables)

A real-valued random variable $X$ with mean $\mu$ is sub-exponential with parameters $(\sigma^{2},b)$ if for all $|t|\leqslant\frac{1}{b}$ we have

[TABLE]

Lemma 2 (Sub-exponential tails)

For $X$ as above we have

[TABLE]

Lemma 3 (Independent sub-exponential random variables)

For independent $X_{i}$ each sub-exponential with parameters $(\sigma_{i}^{2},b_{i})$, the sum $X=\sum_{i}X_{i}$ is sub-exponential with parameters $(\sum_{i}\sigma_{i}^{2},\max_{i}b_{i})$.

2.3 Optimization theory

In this section we very briefly remind some concepts from optimization theory (see for example [FP06] for a reference). Suppose that we want to solve a problem of the form

[TABLE]

where $D\subset\mathbb{R}^{d}$ is an open set. Any point $p$ satisfying the constraints is called feasible. We call the maximizer $p=p^{*}$ the optimal point. The inequality constraint $g_{j}$ is called active at $p$ if $g_{j}(p)=0$. All equality constraints are active at any feasible point.

The optimal point can be characterized by the so called KKT conditions, provided that certain regularity properties are satisfied.

Definition 3 (LICQ constraint qualification)

We say that the LICQ constraint qualification holds, if at the optimal point the gradients of the active constraints are linearly independent.

If the LICQ condition is satisfied, then the optimal point $p=p^{*}$ is a stationary point to the Lagrangian formulated as

[TABLE]

where $\lambda_{i}\in\mathbb{R}$ for $i\in I$ and $\mu_{j}\geqslant 0$ for $j\in J$ are some weights (non-zero only for active constraints). This leads to the so called first order conditions

[TABLE]

In lack of convexity properties, could be that $p^{*}$ is only stationary to $L$, but is not optimal for $L$. Still, the following second order conditions are satisfied

[TABLE]

for all vectors $d$ such that $\frac{\partial g_{j}}{\partial p}(p^{*})\cdot d=0$ for active $j\in J$ and $\frac{\partial h_{i}}{\partial p}(p^{*})\cdot d=0$ for $i\in I$. These vectors are called ”tangent” because they discribe small perturbation of the point that are consisent with the constraints.

2.4 Majorization theory

Definition 4 (Vectors majorization)

For two vectors $u,v\in\mathbb{R}^{d}$ we say that $u$ majorizes $v$, and denote by $u\succ v$, if the following inequalities

[TABLE]

where $u^{\prime}$ and $v^{\prime}$ are vectors with the same components as $u$ and $v$ respectively, sorted in the non-decreasing order.

Definition 5 (Schur convexity)

We say that $f:\mathbb{R}^{d}\rightarrow\mathbb{R}$ is Schur-convex (abbreviated to S-convex) whenever $f(u)\geqslant f(v)$ for any $u,v\in\mathbb{R}^{d}$ such that $u$ majorizes $v$.

Proposition 1 (S-Convexity Criteria [MOA11])

The following statements are true

• •

Every symmetric and convex function is S-convex.

• •

If $f:\mathbb{R}^{d}\rightarrow\mathbb{R}$ is increasing in each coordinate and $h_{i}$ for $i=1,\ldots,d$ are S-convex then the composition $f(h_{1},\ldots,h_{d})$ is S-convex.

3 Main Result

3.0.1 Optimizing the surprise MGF under entropy constraints

Proof (Proof of Theorem 1.1)

We consider (3) for some alphabet $\mathcal{X}$ of fixed size $N$. We will prove that for some constants $\alpha>\beta>0$, the distribution $P_{X}$ optimal to (3) satisfies

[TABLE]

Note that (3) is equivalent to

[TABLE]

over probability vectors $p\in\mathbb{R}^{N}$. Note also that we have to assume that $t\geqslant-1$, as for $t<-1$ the value of (8) is unbounded (because the objective equals $\mathrm{e}^{H(p)}\sum_{i}p_{i}^{1+t}$ which is arbitrarily big whenever one of $p_{i}$ is close to zero). For $t=-1$ we see that the objective in (8) is constant and our statement is trivialy true. Thus we can assume that $t>-1$.

Claim (The optimal solution is not flat)

If $k<\log N$ then the solution $p=p^{*}$ to (9) has at least two different non-zero entries.

Proof (Proof of Claim)

If the optimal solution is a flat distribution, then $H(p^{*})=k$ implies $p^{*}_{i}=2^{-k}$ for all $i$ such that $p^{*}_{i}>0$888In particular $2^{k}$ must be integer, but we don’t use this observation. This means that the value of (8) is $1$. However, by the Jensen inequality applied to a strictly convex function $u\rightarrow\mathrm{e}^{u}$, for any probability vector $q$ such that $q_{i}\not q_{j}\not=0$ for some $i,j$ we have

[TABLE]

In other words, the objective is strictly bigger than $1$ for any non-flat distribution $q$. We conclude that all feasible distributions must be flat.

Now, if $k<\log N$ we consider the probability vector $p$ given by $p_{1}=\delta$, $p_{i}=\frac{1-\delta}{N-1}$ for $i\not=1$, and solve the equation $H(p)=k$. This is equivalent to

[TABLE]

and because the left-hand side equals $\log N$ when $\delta=N^{-1}$ and [math] when $\delta=1$, there is a solution $\frac{1}{N}<\delta<1$ (by continuity and the intermediate value theorem). This solution satisfies $0<p_{1}<p_{2}$ hence is not flat, a contraddiction. ∎

Since for all feasible points $H(p)$ is constant, the objective in (8) can be simplified and the solution is the same as for the program

[TABLE]

over probability vectors $p\in\mathbb{R}^{N}$. In the first step we prove a somewhat weaker result, namely that the optimal solution $p=p^{*}$ for some $0<\alpha<\beta$ satisfies

[TABLE]

Note that we can skipp the zero entries of $p$, as this doesn’t change other constraints $H(p)=k$ and $\sum_{i}p_{i}=1$ neither the objective function. More precisely, if $p^{*}$ is optimal for (9) and $I=\{i:p^{*}_{i}>0\}$ then $p=p^{*}_{i\in I}$ is a local maximizer of

[TABLE]

over vectors $p\in\mathbb{R}^{|I|}$.

Claim (Regularity conditions hold)

If $p=p^{*}$ is optimal to (11), then the LICQ condition is satisfied at $p^{*}$.

Proof (Proof of Claim)

The active constraints are $\sum_{i}p_{i}=1$ and $H(p)=k$. We have $\frac{\partial\sum_{i}p_{i}}{\partial p_{i}}=1$, and $\frac{\partial H(p)}{\partial p_{i}}=\log\frac{1}{p_{i}}-1$. If the gradients are linearly dependent at $p^{*}$ then $\lambda_{1}+\lambda_{2}\left(\log\frac{1}{p^{*}_{i}}-1\right)=0$ for all $i\in I$. If $\lambda_{2}=0$ then $\lambda_{1}=0$ and we are done. If $\lambda_{2}\not=0$ then for some constants $c$ and all $i$ we have $p^{*}_{i}=c$, a contraddiction to the previous claim that $p^{*}$ cannot be flat. ∎

The lagrangian associated with (11) is

[TABLE]

By the first order conditions applied to (12) we (partially) characterize the optimal solution.

Claim (The solution combines two flat distributions)

If $p=p^{*}$ solves (11), then its entries take only two values: $p^{*}_{i}\in\{\alpha,\beta\}$ for some $0<\alpha<\beta$ and all $i\in I$.

Proof (Proof of Claim)

By the first order conditions (justified because of the regularity proven in the last claim) for some $\lambda_{1},\lambda_{2}$ we have

[TABLE]

Note that Equation 13 is equivalent to $g(p^{*}_{i})=c$ for all $i\in I$ and some constant $c$, where $g$ is the function defined by $g(u)=(1+t)u^{t}-\lambda_{1}\log\frac{1}{u}$. The derivative equals $\frac{\partial g}{\partial u}=t(1+t)u^{t-1}+\frac{\lambda_{1}}{u}$, and changes its sign at most once over $u\in(0,1)$. Thus any equation $g(u)=c$ has at most two solutions in $u\in(0,1)$. In particular, since $g(p^{*}_{i})=c$ for all $i$, there are two values $\alpha,\beta$ possible for $p^{*}_{i}$ where $i\in I$. Note that $0\not=\alpha\not=\beta$ as we proved that $p^{*}$ is not uniform. We can assume $\alpha<\beta$. ∎

The last claim is a step towards the characterization Theorem 1.1, but we need to establish that the weight $\alpha$ is used only once and that no zero-weights occur. The numbers $\alpha,\beta$ from the last claim may depend only $t$ and $k$. Below we show that the dependency is basically limited to $k$.

Claim (Optimal probabilities don’t depend on $t$)

For any fixed $k$, there exist finitely many choices for the optimal solution (the choices don’t depend on $t$).

Proof (Proof of Claim)

By last claim, we have

[TABLE]

for some natural numbers $N_{\alpha}=\#\{i:\ p^{*}_{i}=\alpha\},N_{\beta}=\#\{i:p^{*}_{i}=\beta\}$ such that $N_{\alpha}+N_{\beta}=|I|\leqslant N$. Let $g=(g_{1},g_{2}):\mathbb{R}^{2}\rightarrow\mathbb{R}^{2}$ be a function of $\alpha,\beta$ with parameters $N_{\alpha},N_{\beta}$, such that $g_{1}$ and $g_{2}$ are the left-hand sides of the first and second equation of (16) respectively. The jacobian of $g$ with respect to $\alpha,\beta$ equals

[TABLE]

Since the optimal solution is not flat, we have $N_{\alpha},N_{\beta}\not=0$ and $\alpha\not=\beta$. It follows that $\mathrm{det}\frac{\partial g}{\partial(\alpha,\beta)}\not=0$. By the implicit function theorem [KP12], for any $c_{1},c_{2}$ there is at most one solution to $g(\alpha,\beta)=(c_{1},c_{2})$. In particular, setting $c_{1}=1,c_{2}=k$ we conclude that (16) has at most one solution for fixed parameters $N_{\alpha},N_{\beta}$. There are finitely many choices for these paramters and the claim follows. ∎

We will argue that it can be assumed that the hessian matrix of $L$ at the optimal point $p^{*}$ is negatively defined.

Claim (The hessian diagonal is non-zero at the optimal point)

For all but finitely many values of $t$, at the optimal point we have $\frac{\partial^{2}L}{\partial p_{i}^{2}}\not=0$ for all $i$.

Proof (Proof of Claim)

We have

[TABLE]

Therefore if $\frac{\partial^{2}L}{\partial p_{i}^{2}}(p^{*})=0$ for some $i$, then $t(1+t)(p^{*}_{i})^{t-1}+\lambda_{1}(p^{*}_{i})^{-1}=0$. Combining the last claim with (13) and assuming, without loss of generality, that $p^{*}_{i}=\alpha$ ($p^{*}_{i}=\beta$ is analogues) we obtain

[TABLE]

Computing $\lambda_{1}$ from the second and third equation and plugging into the first equation we obtain (note that $\alpha\not=\beta$ is guaranteed)

[TABLE]

This is equivalent to $tc_{1}^{t}+c_{2}=0$ where $\mathrm{poly}(t)$ is a polynomial in $t$ and $c_{1},c_{2}$ are constants depending on $\alpha,\beta$. It follows that $t$ takes only finitely many values for fixed $\alpha,\beta$. Since $\alpha,\beta$ take finitely many values, by the last claim, there are finitely many numbers $t$ such that (23) is satisfied by some $\alpha,\beta$. ∎

Claim (The hessian is negative definite,

or the characterization (10) holds.)

For all but finitely many $t$, the optimal point is as in (10) or makes the hessian of $L$ negative definite.

Proof (Proof of Claim)

Let $t$ be as in the last claim and let $p^{*}$ be optimal for (11). By the second order conditions we have

[TABLE]

Define $N_{\alpha}=\#\{i:p_{i}=\alpha\}$ and $N_{\beta}=\#\{i:p_{i}=\beta\}$. Suppose first that $N_{\alpha},N_{\beta}>1$. Choosing $d_{i_{1}}=\pm\delta,d_{i_{2}}=\mp\delta$ for $i_{1}\not=i_{2}$ such that $p_{i_{1}}=p_{i_{2}}=\alpha$ or $p_{i_{1}}=p_{i_{2}}=\beta$ in (24) and using (19) yields $0\geqslant\left(\frac{\partial^{2}L}{\partial p_{i_{1}}^{2}}(p^{*})+\frac{\partial^{2}L}{\partial p_{i_{2}}^{2}}(p^{*})\right)\cdot\delta^{2}=2\frac{\partial^{2}L}{\partial p_{i_{1}}^{2}}(p^{*})\cdot\delta^{2}$ for all $\delta>0$. Therefore

[TABLE]

By the assumption on $t$ and the previous claim, this implies that $L$ is negative definite at $p^{*}$.

Assume now $N_{\alpha}>1$ but $N_{\beta}=1$. As in the previous part we show that $\frac{\partial^{2}L}{\partial p_{i}^{2}}(p^{*})<0$ for all $i$ such that $p^{*}_{i}=\alpha$. By Equation 19 there exists $c$ such that

[TABLE]

where $c$ depends on $t$ and $\lambda_{1}$. Hence, if $p^{*}_{i_{1}}=\alpha$ and $p^{*}_{i_{2}}=\beta$ then $\frac{\partial^{2}L}{\partial p_{i_{1}}^{2}}(p^{*})<0$ means $\frac{\partial^{2}L}{\partial p_{i_{2}}^{2}}(p^{*})<0$. Therefore $L$ is negative definite at $p^{*}$.

Since we have $N_{\alpha},N_{\beta}\geqslant 1$ (as $p^{*}$ is not a flat distribution) the remaining case is $N_{\alpha}=1$. But this is precisely (10). ∎

Claim (The negative definite hessian implies the characterization (10))

If the hessian of $L$ is negative definite at the optimal point $p^{*}$, then $p^{*}$ satisfies (10).

Proof (Proof of Claim)

Let $p^{*}$ be optimal. Since $\frac{\partial L}{\partial p}(p^{*})=0$ by the first order conditions, $p^{*}$ is a local maximizer of $L$ (with $\lambda_{1},\lambda_{2}$ being fixed parameters). Consider $L$ as a function of $p_{J}=(p_{j})_{j\in J}$ for a fixed subset of indices $J$. Let $c$ be as in (28). By (28) and (19) $L$ is convex for $p\in S^{+}=\cap_{j\in J}\{p_{j}>c\}$ and concave for $p\in S^{-}=\cap_{j\in J}\{p_{j}<c\}$, where $c$ is a constant (depends on $\lambda_{1},t$).

Let $i_{1}\not=i_{2}$ be such that $p^{*}_{i_{1}}=p^{*}_{i_{2}}>c$ for some $i_{1}\not=i_{2}$. Take $J=\{i_{1},i_{2}\}$, fix a positive small number $\delta$ and define $p^{\prime}_{i_{1}}=p^{*}_{i_{1}}+\delta$, $p^{\prime}_{i_{2}}=p^{*}_{i_{2}}-\delta$ and $p^{\prime}_{i}=p^{*}_{i}$ when $i\not\in\{i_{1},\i_{2}\}$. Note that $p^{\prime}$ majorizes $p^{*}$ and $p^{\prime},p^{*}\in S^{+}$ for $\delta$ sufficiently small. Because $L$ is symmetric in variables $\{p_{j}\}_{j\in J}$, by Schur convexity we have $L(p^{\prime})>L(p^{*})$. This shows that there is at most one $i$ such that $p^{*}_{i}>c$.

Similarily, take $i_{1}\not=i_{2}$ such that $p^{*}_{i_{1}}<p^{*}_{i_{2}}<c$. Let $J=\{i_{1},i_{2}\}$, fix a positive small number $\delta$ and define $p^{\prime}_{i_{1}}=p^{*}_{i_{1}}+\delta$, $p^{\prime}_{i_{2}}=p^{*}_{i_{2}}-\delta$ and $p^{\prime}_{i}=p^{*}_{i}$ when $i\not\in\{i_{1},\i_{2}\}$. Note that $p$ is majorized by $p^{*}$ and $p^{\prime},p^{*}\in S^{+}$ for $\delta$ sufficiently small. Because $L$ is symmetric in variables $\{p_{j}\}_{j\in J}$, by Schur convexity we have $L(p^{\prime})>L(p^{*})$. This shows that $p^{*}_{i_{1}}=p^{*}_{i_{2}}$ whenever $p^{*}_{i_{1}},p^{*}_{i_{2}}<c$.

In the first part we established that $\{i:p_{i}=\alpha\}$ for one $i$, the second part implies $p_{j}=\beta$ for $j\not=i$. This finishes the proof of the claim. ∎

The last two claims imply that the solution to (9) is characterized by (10) (for all but finitely many $t$). We now show that probability weights are not zero.

Claim (The optimal point has only positive entries)

If the optimal point $p^{*}$ is as in (10) then it satisfies (7).

Proof (Proof of Claim)

Let $p^{*}\in\mathbb{R}^{N}$ be as in (10). Let $v=\#\{i:p^{*}_{i}>0\}$, and $\delta=\alpha$. Then $p_{i_{0}}=\delta$ and $p_{i}=\frac{1-\delta}{v}$ for some other $v-1$ values of $i$. Moreover we have $\delta>\frac{1-\delta}{v}$. Since $p^{*}$ is not uniform we have $v>2^{k}$. Therefore $(\delta,v)$ solves the following program over $\delta\in(0,1)$ and integers $N\geqslant v>2^{k}$:

[TABLE]

Consider this program under the relaxed assumption that $2^{k}<v\leqslant N$. We show that the maximum is achieved for $v=N$. Indeed if $2^{k}<v<N$ then the gradient of the active constraint is

[TABLE]

and hence satisfies the LICQ condition. The first order conditions yield

The second equation implies $\lambda>0$. The left-hand side of the first equation can be rewritten as $(1+t)\delta^{t}\left(1-\left(\frac{1-\delta}{\delta v}\right)^{t}\right)$ and, because $t,\delta>0$ its sign equals $\mathrm{sgn}\left(1-\frac{1-\delta}{\delta v}\right)$. In turn the sign of the right-hand side equals $\mathrm{sgn}(\lambda)\cdot\mathrm{sgn}\left(\frac{1-\delta}{v\delta}-1\right)$. Note that, because $\lambda>0$, the signs are opposite unless $\frac{1-\delta}{v\delta}=1$. this is not possible as by the assumption on $p^{*}$ we have $\delta\not=\frac{1-\delta}{v}$. This shows that (29) must be maximized at $v=N$, in particular $p^{*}_{i}\not=0$ for all $i$. ∎

3.1 Improved sub-exponential tails

The following lemma parameterizes (6) in terms of the entropy defficiency.

Lemma 4 (Entropy defficiency as a function of bias)

Let $X$ be as in Equation 6. Then the bias $\gamma=\theta-\frac{1}{|\mathcal{X}|}$ and the entropy defficiency $\Delta=\log|\mathcal{X}|-H(X)$ are related as in Table 2.

The proof appears in Appendix 0.A.

Proposition 2 (MGF as the function of bias)

Let $X$ be as in Equation 6. Then

[TABLE]

Lemma 5 (Sub-exponential tails of the surprise)

Let $X$ be as in Equation 6. When $\gamma N=\omega(1)$ then the surprise is sub-exponential with $\sigma^{2}=\gamma\log^{2}(\gamma N)$ and $b=\log(\gamma N)$. When $\gamma N=O(1)$ then the surprise is sub-exponential with $\sigma^{2}=\gamma^{2}N$ and $b=2$.

Proof

Let $M_{j}=\mathbb{E}_{x\sim X}\left(H(X)-\log\frac{1}{P_{X}(x)}\right)^{j}$. Note that $M_{0}=1$ and $M_{1}=0$. By the expansion $\exp(u)=\sum_{j=0}^{\infty}\frac{u^{j}}{j!}$ we obtain

[TABLE]

By Equation 32 we obtain

[TABLE]

and hence

[TABLE]

Note that we also have $M_{j}\leqslant 2^{j}\log^{j}N$ by the proof of LABEL:lemma:tail_no_constraints. Now we split our analysis into the following two cases

Case $\gamma N>2$.

Assume first that $\theta<1-\frac{2}{N}$. Let $\theta=\frac{1}{N}+\gamma$. Then $\frac{\theta N-1}{1-\theta}>2$ and thus

[TABLE]

Moreover $\frac{1}{1-\theta}<\theta N-1$ because of $\theta<1-\frac{2}{N}$. Therefore

[TABLE]

For $\sigma^{2}=\gamma\log^{2}(\gamma N)$, $b=\log(\gamma N)$ and $|t|\leqslant\frac{1}{b}$ we obtain

[TABLE]

which is also valid when $\theta\geqslant 1-\frac{2}{N}$. Note that we need $t\geqslant-1$ in Theorem 1.1, which is automatically satisfied because $b\geqslant 1$.

Case $\gamma N<2$.

We have then $\frac{\theta N-1}{1-\theta}=O(N\gamma)$ and by the Taylor expansion $\log(1+u)=O(u)$ valid for $u=O(1)$ we get

[TABLE]

For $\sigma^{2}=\gamma^{2}N$, $b=\gamma N$ and $|t|\leqslant\frac{1}{b}$. we obtain

[TABLE]

Note that we need $t\geqslant-1$ in Theorem 1.1, for this we can assume $b=\max(\gamma N,1)$.

Having proved the last lemma, we are ready to derive Corollary 1.

Proof

Proof of Corollary 1 We consider two cases

Case $\gamma N=\omega(1)$

By Lemma 4, the assumption $\gamma N=\omega(1)$ is equivalent to $\Delta=\omega(N^{-1})$. Also,

[TABLE]

and

[TABLE]

By Lemma 3, the sum of $n$ such surprises is subexponential with $n\sigma^{2}$ and $b$, hence the tail for $t<\sigma^{2}/b$ is

[TABLE]

Case $\gamma N=O(1)$

By Lemma 4, the assumption $\gamma N=O(1)$ is equivalent to $\Delta=O(N^{-1})$. Also,

[TABLE]

and $\sigma^{2}=\gamma^{2}N=\Delta$. By Lemma 3, the sum of $n$ such surprises is subexponential with $n\sigma^{2}$ and $b$, hence the tail for $t<\sigma^{2}/b$ is

[TABLE]

.

4 Conclusion

We obtained sharp bounds on exponential moments of the surprise when the distribution has a certain (fixed) Shannon entropy. The analysis we did yields a characterization for related extremal problems involving Renyi entropy.

Appendix 0.A Proof of Lemma 4

Proof

Consider the equation

[TABLE]

For $X$ as in Equation 6 we obtain

[TABLE]

which is equivalent to

[TABLE]

Introducing $\delta=\frac{1}{N}+\gamma$, we may rewrite it as

[TABLE]

Case 1: $\gamma N=O(1)$. By the Taylor expansion $\log(1+u)=u+O(u^{2})$ for $u\leqslant 1$ we obtain

[TABLE]

where in the last line we have used the fact that $\gamma=O(1/N)$.

Case 2: $\gamma N=\omega(1)$. Multiplying both sides of Equation 35 by $N$, and using the assumption we obtain

[TABLE]

therefore

[TABLE]

This finishes the proof ∎.

Appendix 0.B Codes