The Capacity of Count-Constrained ICI-Free Systems

TL;DR

This paper uses a Markov chain approach to determine the capacity of a broad class of q-ary ICI-free constrained systems with arbitrary count constraints, advancing understanding of their information limits.

Contribution

It introduces a novel Markov chain method to analyze the capacity of count-constrained ICI-free systems, generalizing previous models.

Findings

Derived capacity formulas for various count constraints

Extended analysis to general q-ary systems

Provided insights into the limits of ICI-free constrained channels

Abstract

A Markov chain approach is applied to determine the capacity of a general class of q-ary ICI-free constrained systems that satisfy an arbitrary count constraint.

Equations

$$\mathsf{cap}({\mathsf{S}}_{q;a,b},\mathbf{p}):=\underset{\epsilon \to 0^{+}}{\lim }\underset{n\to \infty }{\mathrm{limsup}}\frac{1}{n}\log |{\mathsf{S}}_{q;a,b}(\mathbf{p},\epsilon )\cap {\Sigma }^n|.$$

$$\mathsf{cap}({\mathsf{S}}_{q;a,b}):=\underset{n\to \infty }{\lim }\frac{1}{n}\log |{\mathsf{S}}_{q;a,b}\cap {\Sigma }^n|.$$

$$\sum _{e\in E_{\mathrm{in}}(v)}P(e)=\sum _{e\in E_{\mathrm{out}}(v)}P(e).$$

$${\mathsf{H}}(P):=-\sum_{e\in E}P(e)\log P(e)-\biggl{(}-\sum_{v\in V}\pi(v)\log\pi(v)\biggr{)}\;.$$

$${\mathbb{E}}_P(\mathbf{f})=\sum _{e\in E}P(e)\mathbf{f}(e).$$

$${\mathcal{I}}_{s}(e)=\left\{\begin{array}[]{lcl}1&&\textrm{if ${\mathsf{L}}(e)=s$}\\ 0&&\textrm{otherwise}\end{array}\right.\;.$$

$$\mathsf{cap}({\mathsf{S}}_{q;a,b},\mathbf{p})=\underset{\genfrac{}{}{0ex}{}{P\in \Delta (G_{q;a,b}):}{{\mathbb{E}}_P({\mathcal{I}}_W)={\mathbf{p}}^{\prime }}}{\sup }\mathsf{H}(P),$$

$$\mathsf{cap}({\mathsf{S}}_{q;a,b},\mathbf{p})=\underset{\genfrac{}{}{0ex}{}{P\in \Delta (G_{q;a,b}):}{{\mathbb{E}}_P({\mathcal{I}}_{\Sigma })=\mathbf{p}}}{\sup }\mathsf{H}(P).$$

$$\underset{P\in {\Delta }_{\mathbf{p},\epsilon }}{\sup }\mathsf{H}(P)=\underset{P\in \overline{{\Delta }_{\mathbf{p},\epsilon }}}{\sup }\mathsf{H}(P),$$

$$\underset{\epsilon \to 0^{+}}{\lim }\underset{P\in \overline{{\Delta }_{\mathbf{p},\epsilon }}}{\sup }\mathsf{H}(P)=\underset{P\in {\Delta }_{\mathbf{p},0}}{\sup }\mathsf{H}(P).$$

$$\underset{\epsilon \to 0^{+}}{\lim }\underset{P\in \overline{{\Delta }_{\mathbf{p},\epsilon }}}{\sup }\mathsf{H}(P)=\underset{\epsilon \to 0^{+}}{\lim }\mathsf{H}(P_{\epsilon })=\mathsf{H}(P_0)\le \underset{P\in {\Delta }_{\mathbf{p},0}}{\sup }\mathsf{H}(P),$$

$$\Bigl{(}\,A_{G;{\mathbf{f}}}({\mathbf{\xi}})\,\Bigr{)}_{u,v}=\sum_{e\in E_{\mathrm{out}}(u)\cap E_{\mathrm{in}}(v)}2^{-{\mathbf{\xi}}\cdot{\mathbf{f}}(e)}\;.$$

$$\underset{\genfrac{}{}{0ex}{}{P\in \Delta (G):}{{\mathbb{E}}_P(\mathbf{f})=\mathbf{r}}}{\sup }\mathsf{H}(P)=\underset{\xi \in {\mathbb{R}}^t}{\inf }\left\{\xi \cdot \mathbf{r}+\log \lambda (A_{G;\mathbf{f}}(\xi ))\right\}.$$

$$\mathsf{cap}({\mathsf{S}}_{3;1,1},\rho )$$

$$A_{G;({\mathcal{I}}_{I},{\mathcal{I}}_{H})}(\xi_{I},\xi_{H})=\left(\begin{array}[]{ccc}2^{-\xi_{H}}&1&2^{-\xi_{I}}\\ 0&0&1+2^{-\xi_{I}}\\ 2^{-\xi_{H}}&0&1+2^{-\xi_{I}}\end{array}\right)\;.$$

$$\mathsf{cap}({\mathsf{S}}_{3;1,1},\rho )=\log \left(\underset{y,z\in (0,\infty {)}^2}{\inf }\frac{\lambda (A(y,z))}{y^{{\rho }_I}{z}^{{\rho }_H}}\right),$$

$$A(y,z):=\left(\begin{array}[]{ccc}z&1&y\\ 0&0&1+y\\ z&0&1+y\end{array}\right)\;.$$

$$z=z(y,\lambda )=\frac{{\lambda }^2(\lambda -y-1)}{{\lambda }^2-\lambda +y+1}.$$

$$\mathsf{cap}({\mathsf{S}}_{3;1,1},\rho )=\log \left(\underset{(y,\lambda )\in \mathcal{U}}{\inf }g(y,\lambda )\right),$$

$$g(y,\lambda )=\frac{\lambda }{y^{{\rho }_I}}{\left(\frac{{\lambda }^2-\lambda +y+1}{{\lambda }^2(\lambda -y-1)}\right)}^{{\rho }_H}$$

$$\mathcal{U}=\{(y,\lambda )\in {\mathbb{R}}^2:\lambda >y+1>1\}.$$

$$\mathsf{cap}({\mathsf{S}}_{3;1,1},\rho )=\log \left[\frac{\lambda }{y^{{\rho }_I}}{\left(\frac{{\lambda }^2-\lambda +y+1}{{\lambda }^2(\lambda -y-1)}\right)}^{{\rho }_H}\right],$$

$$y=-1-2\tau +2\sqrt{1+\tau +{\tau }^2}\text{mbox}and\lambda =2,$$

$$y=\frac{{\rho }_I(\lambda -2)}{1-2{\rho }_L}$$

$$\displaystyle Z(x):=(1-2\rho_{L})\rho_{L}x^{3}+\bigl{(}(\rho_{L}-\rho_{H})^{2}-(1-2\rho_{L})\bigr{)}x^{2}$$

$$\mathsf{cap}({\mathsf{S}}_{q;a,b},\mathbf{p})=\mathsf{cap}({\mathsf{S}}_{3;1,1},\rho )+\mathsf{h}(\mathbf{p})-\mathsf{h}(\rho ),$$

$$\sum _{u\in V}\mathsf{P}(u,s)=p_s,\text{for all}s\in \Sigma .$$

$$\mathsf{Q}(u,X)=\sum _{s\in X}\mathsf{P}(u,s).$$

$${\mathbb{E}}_{\mathsf{Q}}({\mathcal{I}}_X)=\sum _{u\in V}\mathsf{Q}(u,X)=\sum _{s\in X}\sum _{u\in V}\mathsf{P}(u,s).$$

$${\mathbb{E}}_{\mathsf{Q}}({\mathcal{I}}_X)=\sum _{s\in X}{p}_s={\rho }_X,\text{for all}X\in \{L,I,H\}.$$

Full text

I Introduction

Let $\Sigma$ be an alphabet of a finite size $q\geq 2$. A word over $\Sigma$ is any finite string ${\mathbf{w}}=w_{1}w_{2}\ldots w_{n}$ where $w_{i}\in\Sigma$. Let ${\mathcal{F}}$ be a finite set of words over $\Sigma$. The (finite-type) constrained system ${\mathsf{S}}_{\mathcal{F}}$ consists of all words ${\mathbf{w}}=w_{1}w_{2}\ldots w_{n}$ over  $\Sigma$ such that ${\mathcal{F}}$ contains none of their substrings $w_{i}w_{i+1}\ldots w_{j}$, for any $1\leq i\leq j\leq n$. We refer to the set ${\mathcal{F}}$ as the set of forbidden words defining the constrained system ${\mathsf{S}}_{\mathcal{F}}$. The constrained system ${\mathsf{S}}_{\mathcal{F}}$ can be presented by a (finite) directed edge-labeled graph $G$, with edges labeled with symbols from $\Sigma$, such that ${\mathsf{S}}_{\mathcal{F}}$ is the set of all words obtained by reading off the labels along paths of $G$. For a proof of this fact, we refer the reader to [4], which provides a comprehensive introduction to the subject of constrained systems.

Our specific interest is in a general class of “inter-cell interference free” (in short, “ICI-free”) constrained systems, which we now define. For prescribed positive integers $a$, $b$, and $q$ such that $a+b\leq q$, let $\Sigma$ be an alphabet of size $q$ which is assumed to be partitioned into three (disjoint) subsets $L$, $H$, and $I$, of sizes $a$, $b$, and $q-a-b$, respectively. The elements in $L$ (respectively, $H$) represent the “low” (respectively, “high”) symbols of $\Sigma$, while those in $I$ are the “intermediate” symbols. The ICI-free constrained system that we consider is the constrained system111Since only the sizes of $\Sigma$, $L$, and $H$ will matter, we identify the constrained system by the sizes of these sets. ${\mathsf{S}}_{q;a,b}:={\mathsf{S}}_{{\mathcal{F}}_{q;a,b}}$ defined by the set of forbidden words ${\mathcal{F}}_{q;a,b}:=\{w_{1}w_{2}w_{3}:w_{1},w_{3}\in H,w_{2}\in L\}$. A graph $G_{q;a,b}$ presenting the constrained system ${\mathsf{S}}_{q;a,b}$ is shown in Fig. 1.

We additionally impose a count constraint defined by a given probability vector ${\mathbf{p}}=(p_{s})_{s\in\Sigma}$ (with nonzero entries that sum to $1$), which specifies the frequencies of occurrence of each $s\in\Sigma$ within words belonging to ${\mathsf{S}}_{q;a,b}$. To avoid trivialities, we will assume $\rho_{L}:=\sum_{s\in L}p_{s}$ and $\rho_{H}:=\sum_{s\in H}p_{s}$ to be strictly positive. The probability $\rho_{I}:=\sum_{s\in I}p_{s}$ is allowed to be [math].

For $\varepsilon>0$, let ${\mathsf{S}}_{q;a,b}({\mathbf{p}},\varepsilon)$ denote the subset of ${\mathsf{S}}_{q;a,b}$ consisting of all words ${\mathbf{w}}\in{\mathsf{S}}_{q;a,b}$ in which the number of occurrences of each symbol $s\in\Sigma$ lies in the interval \bigl{(}(p_{s}-\varepsilon)|{\mathbf{w}}|,(p_{s}+\varepsilon)|{\mathbf{w}}|\bigr{)}, where $|{\mathbf{w}}|$ denotes the length of ${\mathbf{w}}$. The capacity (or the asymptotic information rate) of ${\mathsf{S}}_{q;a,b}$ under the count constraint specified by ${\mathbf{p}}$ is defined as222All logarithms in this work are to the base $2$.

[TABLE]

This quantifies, for large $n$, the exponential rate of growth of the number of length-$n$ words in ${\mathsf{S}}_{q;a,b}$ in which the relative frequency of occurrence of each symbol $s\in\Sigma$ is approximately $p_{s}$. Dropping the count constraint, we also define the (ordinary) capacity of the constrained system ${\mathsf{S}}_{q;a,b}$ to be333By a standard sub-additivity argument, the limit in this definition exists.

[TABLE]

The quantities ${\mathsf{cap}}({\mathsf{S}}_{q;a,b})$ and ${\mathsf{cap}}({\mathsf{S}}_{q;a,b},{\mathbf{p}})$ were studied in [1], [5], [6], motivated by proposed coding schemes to mitigate inter-cell interference in flash memory devices.444These references used a different definition of ${\mathsf{cap}}({\mathsf{S}}_{q;a,b},{\mathbf{p}})$, which is shown in Appendix A to be equivalent to our definition in (1). Using standard techniques from the theory of constrained systems (see e.g., [4]) the (ordinary) capacity ${\mathsf{cap}}({\mathsf{S}}_{q;a,b})$ was shown in [1] to be the largest real root of the cubic polynomial $x^{3}-qx^{2}+abx-ab(q-b)$. The analysis of ${\mathsf{cap}}({\mathsf{S}}_{q;a,b},{\mathbf{p}})$ in [1] is based on combinatorial arguments, and a Stirling approximation of the resulting expressions then yields a bivariate function which needs to be maximized (numerically) in order to obtain the values of ${\mathsf{cap}}({\mathsf{S}}_{q;a,b},{\mathbf{p}})$.

In this work, we make use of a result from [3] to formulate the problem of determining the capacity ${\mathsf{cap}}({\mathsf{S}}_{q;a,b},{\mathbf{p}})$ as an optimization problem over Markov chains defined on the graph $G_{q;a,b}$ shown in Fig. 1. By shifting to the dual optimization problem, we then derive an analytical solution to this optimization problem, which results in an exact expression for ${\mathsf{cap}}({\mathsf{S}}_{q;a,b},{\mathbf{p}})$ given in Theorems 3 and 4 in Section III. While our analysis is tailored to count-constrained ICI-free systems, some of the tools that we use may be applicable to other constrained systems as well (see [2]).

II Markov Chains and Optimization

Let $G=(V,E)$ be a directed graph with vertex set $V$ and (directed) edge set $E$. For a vertex $v\in V$, we let $E_{\mathrm{in}}(v)$ and $E_{\mathrm{out}}(v)$ denote the set of incoming and outgoing edges, respectively, incident with $v$.

A stationary Markov chain on $G$ is a probability distribution P=\bigl{(}P(e)\bigr{)}_{e\in E} on $E$, with the property that for each $v\in V$, the sum of the probabilities on the incoming edges of $v$ is equal to that on the outgoing edges of $v$:

[TABLE]

The induced stationary distribution on the vertex set $V$ is given by $\pi(v)=\sum_{e\in E_{\mathrm{out}}(v)}P(e)$, for all $v\in V$. The set of all stationary Markov chains on $G$ is denoted by $\Delta(G)$.

The entropy rate of a stationary Markov chain $P$ on $G$ is defined as

[TABLE]

Since ${\mathsf{H}}(P)=-\sum_{v\in V}\sum_{e\in E_{\mathrm{out}}(v)}P(e)\log(P(e)/\pi(v))$, the convexity properties of relative entropy imply that $P\mapsto{\mathsf{H}}(P)$ is a concave function.

Given a Markov chain $P\in\Delta(G)$, along with a vector of real-valued functions ${\mathbf{f}}=(f_{1}\;f_{2}\;\ldots\;f_{t}):E\rightarrow{\mathbb{R}}^{t}$, we denote by ${\mathbb{E}}_{P}({\mathbf{f}})$ the expected value of ${\mathbf{f}}$ with respect to $P$:

[TABLE]

We will only need the following special case of the function ${\mathbf{f}}$. Let ${\mathsf{L}}:E\to\Sigma$ be a labeling of the edges of the graph $G$ with symbols from $\Sigma$. For a subset $W$ of $\Sigma$ of size $t$, we define the vector indicator function ${\mathcal{I}}_{W}:E\rightarrow{\mathbb{R}}^{t}$ by ${\mathcal{I}}_{W}=({\mathcal{I}}_{s})_{s\in W}$, where ${\mathcal{I}}_{s}:E\rightarrow{\mathbb{R}}$ is the indicator function for a symbol $s\in\Sigma$:

[TABLE]

Then, ${\mathbb{E}}_{P}({\mathcal{I}}_{W})$ is a vector in ${\mathbb{R}}^{t}$ whose entry that is indexed by $s\in W$ is the probability that an edge chosen according to the distribution $P$ is labeled with the symbol $s$.

These definitions allow us to state the following result, which expresses ${\mathsf{cap}}({\mathsf{S}}_{q;a,b},{\mathbf{p}})$ as the solution to a convex optimization problem.

Proposition 1**.**

We have

[TABLE]

for any $W\subset\Sigma$ of size $q-1$ and ${\mathbf{p}}^{\prime}=(p_{s})_{s\in W}$.

Proof.

As a consequence of [3, Lemma 2], for any $\varepsilon>0$, $\limsup_{n\to\infty}(1/n)\log|{\mathsf{S}}_{q;a,b}({\mathbf{p}},\varepsilon)\cap\Sigma^{n}|$ is equal to $\sup{\mathsf{H}}(P)$, the supremum being over stationary Markov chains $P\in\Delta(G_{q;a,b})$ such that ${\mathbb{E}}_{P}({\mathcal{I}}_{\Sigma})\in({\mathbf{p}}-\varepsilon\cdot{\mathbf{1}},{\mathbf{p}}+\varepsilon\cdot{\mathbf{1}})$ (where ${\mathbf{1}}$ denotes the all-one vector in ${\mathbb{R}}^{q}$). We claim that as $\varepsilon\to 0$, these suprema converge to $\sup{\mathsf{H}}(P)$, the supremum now being over stationary Markov chains $P\in\Delta(G_{q;a,b})$ such that ${\mathbb{E}}_{P}({\mathcal{I}}_{\Sigma})={\mathbf{p}}$. With this, we would have

[TABLE]

The constraint ${\mathbb{E}}_{P}({\mathcal{I}}_{\Sigma})={\mathbf{p}}$ in the supremum on the right-hand side (RHS) above can be replaced by ${\mathbb{E}}_{P}({\mathcal{I}}_{W})=(p_{s})_{s\in W}$, since the latter implies ${\mathbb{E}}_{P}({\mathcal{I}}_{\{s\}})=p_{s}$ for the remaining symbol $s\in\Sigma\setminus W$. This would prove the proposition.

We now prove the claim above. To this end, for $\varepsilon>0$, define $\Delta_{{\mathbf{p}},\varepsilon}$ to be the set of all stationary Markov chains $P\in\Delta(G_{q;a,b})$ such that ${\mathbb{E}}_{P}({\mathcal{I}}_{\Sigma})\in({\mathbf{p}}-\varepsilon\cdot{\mathbf{1}},{\mathbf{p}}+\varepsilon\cdot{\mathbf{1}})$. Its closure $\overline{\Delta_{{\mathbf{p}},\varepsilon}}$ is the set of all $P\in\Delta(G_{q;a,b})$ such that ${\mathbb{E}}_{P}({\mathcal{I}}_{\Sigma})\in[{\mathbf{p}}-\varepsilon\cdot{\mathbf{1}},{\mathbf{p}}+\varepsilon\cdot{\mathbf{1}}]$. By continuity of the mapping $P\mapsto{\mathsf{H}}(P)$, we have

[TABLE]

and the latter supremum is in fact a maximum. Finally, let $\Delta_{{\mathbf{p}},0}$ denote the set of all $P\in\Delta(G_{q;a,b})$ such that ${\mathbb{E}}_{P}({\mathcal{I}}_{\Sigma})={\mathbf{p}}$. We wish to show that

[TABLE]

The limit on the left-hand side (LHS) of (5) exists since $\sup_{P\in\overline{\Delta_{{\mathbf{p}},\varepsilon}}}{\mathsf{H}}(P)$ is a monotone function of $\varepsilon$.

Since $\Delta_{{\mathbf{p}},0}\subseteq\overline{\Delta_{{\mathbf{p}},\varepsilon}}$ for all $\varepsilon>0$, the RHS above cannot exceed the LHS. To prove the reverse inequality, suppose that $P_{\varepsilon}$ achieves the supremum over $P\in\overline{\Delta_{{\mathbf{p}},\varepsilon}}$. Passing to a subsequence if necessary, $P_{\varepsilon}$ converges (as $\varepsilon\to 0^{+}$) to some $P_{0}\in\Delta(G_{q;a,b})$. From the fact that ${\mathbb{E}}_{P}({\mathcal{I}}_{\Sigma})$ is continuous in $P$, it follows that $P_{0}\in\Delta_{{\mathbf{p}},0}$. Hence, again via the continuity of the mapping $P\mapsto{\mathsf{H}}(P)$, we obtain

[TABLE]

which proves our claim. ∎

Thus, computation of the quantity ${\mathsf{cap}}({\mathsf{S}}_{q;a,b},{\mathbf{p}})$ requires the solution of a constrained optimization problem in which the objective function $P\mapsto{\mathsf{H}}(P)$ is concave, and the constraints are linear. The theory of convex duality based upon Lagrange multipliers provides a method to translate the problem into an unconstrained optimization with a convex objective function [3].

In order to reformulate the problem, we need to introduce a vector-valued matrix function that generalizes the adjacency matrix of a directed graph $G=(V,E)$. For a function ${\mathbf{f}}:E\rightarrow{\mathbb{R}}^{t}$ and ${\mathbf{\xi}}\in{\mathbb{R}}^{t}$, let $A_{G;{\mathbf{f}}}({\mathbf{\xi}})$ be the matrix defined by

[TABLE]

We remark that for any function ${\mathbf{f}}$, the matrix $A_{G;{\mathbf{f}}}({\mathbf{0}})$ is precisely the adjacency matrix of $G$. Moreover, for any choice of ${\mathbf{\xi}}\in{\mathbb{R}}^{t}$, the matrix $A_{G;{\mathbf{f}}}({\mathbf{\xi}})$ is (entry-wise) non-negative, so that it has a unique largest positive eigenvalue, called the Perron eigenvalue, which we denote by $\lambda(A_{G;{\mathbf{f}}}({\mathbf{\xi}}))$.

The following lemma is the main tool in translating the constrained optimization problem to a more tractable form. It is a consequence of standard results in the theory of convex duality.

Lemma 2**.**

Let $G$ and ${\mathbf{f}}$ be as above. Then, for any ${\mathbf{r}}\in{\mathbb{R}}^{t}$,

[TABLE]

Note that since $P\mapsto{\mathsf{H}}(P)$ is a concave function, by convex duality, the objective function on the RHS of the lemma is a convex function of ${\mathbf{\xi}}$. Moreover, it is a differentiable function of $\xi$ whenever the graph $G$ is strongly-connected (as is the case when $G=G_{q;a,b}$): the matrix $A_{G;{\mathbf{f}}}({\mathbf{\xi}})$ is then irreducible for all ${\mathbf{\xi}}\in{\mathbb{R}}^{t}$, so that its Perron eigenvalue is simple, and hence differentiable as a function of ${\mathbf{\xi}}$. Consequently, the objective function can be minimized by identifying the point at which its gradient with respect to ${\mathbf{\xi}}$ vanishes.

We illustrate the use of Proposition 1 and Lemma 2 to determine ${\mathsf{cap}}({\mathsf{S}}_{q;a,b},{\mathbf{p}})$ in the case of $q=3$ in Section III-A. We will later show in Section III-B that the general $q\geq 3$ case can be reduced to $q=3$.

III Computation of ${\mathsf{cap}}({\mathsf{S}}_{q;a,b},{\mathbf{p}})$

The simplest case is that of $q=2$, i.e., the ${\mathsf{S}}_{2;1,1}$ constrained system. This is the “no-$101$” constrained system, which forbids the occurrence of the string $101$. The value of ${\mathsf{cap}}({\mathsf{S}}_{2;1,1},(1{-}p,p))$, for $p\in(0,1)$, can be computed via Proposition 1 and Lemma 2, using an analysis similar to (but simpler than) that in Section III-A. However, we do not provide the details of this analysis, as it is not difficult to convince oneself that ${\mathsf{cap}}({\mathsf{S}}_{2;1,1},(1{-}p,p))={\mathsf{cap}}({\mathsf{S}}_{3;1,1},(1{-}p,0,p))$. Thus, we start with the $q=3$ case.

III-A The Case $q=3$ and $a=b=1$

The key to our analysis of the capacity ${\mathsf{cap}}({\mathsf{S}}_{q;a,b},{\mathbf{p}})$ is the case $(q;a,b)=(3;1,1)$. As noted above, this case subsumes the case $(q;a,b)=(2;1,1)$. Moreover, as we will show in the next subsection, the computation of ${\mathsf{cap}}({\mathsf{S}}_{q;a,b},{\mathbf{p}})$ for any $q\geq 3$, $a\geq 1$, and $b\geq 1$ can be reduced to the problem of computing ${\mathsf{cap}}({\mathsf{S}}_{3;1,1},{\mathbf{\rho}})$, where the entries of ${\mathbf{\rho}}$ are $\rho_{X}=\sum_{s\in X}p_{s}$, for $X\in\{L,I,H\}$.

So, consider a ternary alphabet $\Sigma$ partitioned into singleton subsets $L$, $I$, and $H$. By abuse of notation, we will assume that $L$, $I$, and $H$ are the actual elements of the alphabet $\Sigma$. The graph presentation of ${\mathsf{S}}_{3;1,1}$ is given by Fig. 1, regarding each arrowed line in the figure as a single edge.

Let the count constraint vector be ${\mathbf{\rho}}=(\rho_{L},\rho_{I},\rho_{H})$, with $\rho_{L},\rho_{H}\in(0,1)$ and $\rho_{I}\in[0,1)$. From Proposition 1 and Lemma 2 (applied with ${\mathbf{f}}=({\mathcal{I}}_{I},{\mathcal{I}}_{H})$), we obtain

[TABLE]

where

[TABLE]

As noted after Lemma 2, the objective function on the RHS of (LABEL:eq:R3p:1) can be minimized by identifying the point at which its gradient with respect to $(\xi_{I},\xi_{H})$ equals [math].

The case $\rho_{I}=0$ needs a little extra care, as in this case the infimum in (LABEL:eq:R3p:1) is achieved by letting $\xi_{I}\to\infty$. This follows from the fact that for any fixed $\xi_{H}$, the Perron eigenvalue $\lambda(A_{G;({\mathcal{I}}_{I},{\mathcal{I}}_{H})}(\xi_{I},\xi_{H}))$ is strictly decreasing in $\xi_{I}$ (see Problem 3.12 in [4]). Thus, the RHS of (LABEL:eq:R3p:1) reduces to the single-variable optimization problem $\inf_{\xi_{H}}\{\rho_{H}\xi_{H}+\log\lambda(A_{G;{\mathcal{I}}_{H}}(\xi_{H}))\}$, where $A_{G;{\mathcal{I}}_{H}}(\xi_{H})$ is the matrix obtained by setting $2^{-\xi_{I}}=0$ in (7).

We first assume that $\rho_{I}>0$ (describing later the minor modifications to be made to handle the case $\rho_{I}=0$). We make the change of variables $y=2^{-\xi_{I}}$ and $z=2^{-\xi_{H}}$ to get

[TABLE]

where $\lambda(A(y,z))$ is the Perron eigenvalue of the matrix

[TABLE]

It is easily checked that the determinant of the Jacobian of the transformation $(\xi_{I},\xi_{H})\mapsto(y,z)$ is nonzero for all $(\xi_{I},\xi_{H})\in{\mathbb{R}}^{2}$. It follows from this that for any $(\xi_{I},\xi_{H})\in{\mathbb{R}}^{2}$, the gradient of the objective function in (LABEL:eq:R3p:1) is [math] at $(\xi_{I},\xi_{H})$ if and only if the gradient of the objective function in (8) is [math] at $(y,z)=(2^{-\xi_{I}},2^{-\xi_{H}})$. Thus, the minimization in (8) can be carried out by identifying the positive values of $y,z$ at which the gradient of $\lambda(A(y,z))/(y^{\rho_{I}}z^{\rho_{H}})$ vanishes.

To do this, we make another convenient change of variables: $(y,z)\mapsto(y,\lambda)$ with $\lambda=\lambda(A(y,z))$. This mapping is invertible: since $A(y,z)$ is irreducible for $y,z>0$, it follows from Problem 3.12 in [4] that $\lambda(A(y,z))$ is strictly increasing in $z$ for every fixed $y>0$. Also, for each fixed $y>0$, the mapping $z\mapsto\lambda(A(y,z))$ is a continuous function from $(0,\infty)$ onto $(y+1,\infty)$ (as it is easy to see that $\lambda(A(y,0))=y+1$). For every fixed $y>0$, the inverse mapping $(y,\lambda)\mapsto(y,z)$ is determined by setting the characteristic polynomial of $A(y,z)$ equal to [math], and is given by

[TABLE]

It can be verified by direct computation555Also see Appendix B for an argument using Perron–Frobenius theory. that $\partial z/\partial\lambda>0$ whenever $\lambda>y+1>1$, and hence, the Jacobian determinant of the transformation $(y,z)\mapsto(y,\lambda)$ is nonzero for all $y,z>0$. From this, arguing as above for the mapping $(\xi_{I},\xi_{H})\mapsto(y,z)$, we obtain via (8) and (9) that

[TABLE]

where

[TABLE]

and

[TABLE]

Moreover, the infimum in (10) is obtained at any point $(y,\lambda)\in{\mathcal{U}}$ where the partial derivatives of $g(y,\lambda)$ vanish.

Turning now to the case $\rho_{I}=0$, it can be handled by setting $y=0$ in the discussion above, assuming the convention that $0^{0}=1$. Thus, the RHS of (8) reduces to $\log\left(\inf_{z>0}\lambda(A(0,z))/z^{\rho_{H}}\right)$, and the RHS of (10) becomes $\log\left(\inf_{\lambda>1}g_{0}(\lambda)\right)$, where $g_{0}(\lambda):=g(0,\lambda)$. This latter infimum is achieved at any point $\lambda>1$ where the derivative $g_{0}^{\prime}(\lambda)$ equals [math].

In Appendix C, we compute the partial derivatives $\partial g/\partial y$ and $\partial g/\partial\lambda$, each being a cubic multinomial in $y$ and $\lambda$. We then find explicitly their common root, thereby yielding the following result.

Theorem 3**.**

For ${\mathbf{\rho}}=(\rho_{L},\rho_{I},\rho_{H})\in(0,1)\times[0,1)\times(0,1)$:

[TABLE]

where $(y,\lambda)$ is given as follows.

• •

If $\rho_{L}={\frac{1}{2}}$ and $\rho_{I}=0$, then $y=0$ and $\lambda=2$.

• •

If $\rho_{L}={\frac{1}{2}}$ and $\rho_{I}>0$, then

[TABLE]

where $\tau:=\rho_{H}/\rho_{I}$.

• •

If $\rho_{L}\neq{\frac{1}{2}}$, then

[TABLE]

and $\lambda$ is a root of the cubic polynomial

[TABLE]

chosen as follows: if $\rho_{L}<{\frac{1}{2}}$, then $\lambda$ is the largest real (positive) root of $Z(x)$; and if $\rho_{L}>{\frac{1}{2}}$, then $\lambda$ is the smallest real (positive) root of $Z(x)$.

It is worth noting that for ${\mathbf{\rho}}=({\frac{1}{2}},0,{\frac{1}{2}})$ we obtain ${\mathsf{cap}}({\mathsf{S}}_{3;1,1},({\frac{1}{2}},0,{\frac{1}{2}}))={\frac{1}{2}}\log 3$. Thus, ${\mathsf{cap}}({\mathsf{S}}_{2;1,1},({\frac{1}{2}},{\frac{1}{2}}))={\frac{1}{2}}\log 3$, which agrees with the rate derived (using two different approaches) in [5].

III-B The General Case of $q\geq 3$

Consider now a constrained system ${\mathsf{S}}_{q;a,b}$ over a $q$-ary alphabet $\Sigma$ for some $q\geq 3$, where $\Sigma$ is partitioned into the subsets $L$, $H$, and $I$ of sizes $a\geq 1$, $b\geq 1$, and $q-a-b\geq 0$, respectively. Let ${\mathbf{p}}=(p_{s})_{s\in\Sigma}$ be a given count constraint vector, and define $\rho_{X}=\sum_{s\in X}p_{s}$ for $X\in\{L,I,H\}$. If $I=\emptyset$, we set $\rho_{I}=0$. The probabilities $\rho_{L}$ and $\rho_{H}$ are assumed to be strictly positive.

The aim of this subsection is to prove the result stated next. The statement requires the following standard definition: the entropy of a probability vector ${\mathbf{u}}=(u_{i})_{i}$ is defined as ${\mathsf{h}}({\mathbf{u}})=-\sum_{i}u_{i}\log u_{i}$.

Theorem 4**.**

For ${\mathsf{S}}_{q;a,b}$ and ${\mathbf{p}}$ as above:

[TABLE]

where the entries of ${\mathbf{\rho}}$ are $\rho_{X}=\sum_{s\in X}p_{s}$, for $X\in\{L,I,H\}$.

Thus, the computation of ${\mathsf{cap}}({\mathsf{S}}_{q;a,b},{\mathbf{p}})$ reduces to the problem of computing ${\mathsf{cap}}({\mathsf{S}}_{3;1,1},{\mathbf{\rho}})$, which was solved explicitly in Theorem 3. The rest of this subsection is devoted to a proof of Theorem 4.

Let ${\mathsf{P}}={({\mathsf{P}}(u,s))}_{u,s}$ be a stationary Markov chain on the labeled graph $G_{q;a,b}$ on the vertex set $V=\{1,2,3\}$ in Fig. 1, where ${\mathsf{P}}(u,s)$ is the probability of the edge labeled $s$ leaving vertex $u$ (and ${\mathsf{P}}(u,s)\equiv 0$ if there is no such edge). Note that for any $s\in\Sigma$, we have ${\mathbb{E}}_{{\mathsf{P}}}({\mathcal{I}}_{s})=\sum_{u\in V}{\mathsf{P}}(u,s)$. Thus, the constraint ${\mathbb{E}}_{{\mathsf{P}}}({\mathcal{I}}_{\Sigma})={\mathbf{p}}$ on the RHS of (4) is equivalently expressed as

[TABLE]

Now, for $u\in V$ and $X\in\{L,I,H\}$, define

[TABLE]

Note that ${\mathsf{Q}}={({\mathsf{Q}}(u,X))}_{u,X}$ is a stationary Markov chain on the graph in Fig. 1, where each arrowed line in the figure is regarded as a single edge (this graph is $G_{3;1,1}$). Moreover, we have for $X\in\{L,I,H\}$,

[TABLE]

Thus, if we impose the constraint (11) on the Markov chain ${\mathsf{P}}$, we obtain

[TABLE]

In other words, the constraint ${\mathbb{E}}_{{\mathsf{P}}}({\mathcal{I}}_{\Sigma})={\mathbf{p}}$ on the Markov chain ${\mathsf{P}}$ induces the constraint ${\mathbb{E}}_{{\mathsf{Q}}}({\mathcal{I}}_{\{L,I,H\}})={\mathbf{\rho}}$ on the Markov chain ${\mathsf{Q}}$. Finally, observe that ${\mathsf{P}}$ and ${\mathsf{Q}}$ induce the same stationary distribution on $V$:

[TABLE]

The following lemma is the key to proving Theorem 4.

Lemma 5**.**

For a Markov chain ${\mathsf{P}}\in\Delta(G_{q;a,b})$ with ${\mathbb{E}}_{{\mathsf{P}}}({\mathcal{I}}_{\Sigma})={\mathbf{p}}$, and ${\mathsf{Q}}\in\Delta(G_{3;1,1})$ as above, we have

[TABLE]

with equality holding if and only if ${\mathsf{P}}(u,s)=(p_{s}/\rho_{X}){\mathsf{Q}}(u,X)$ for every $s\in X$ (where ${\mathsf{P}}(u,s)=0$ when $p_{s}=\rho_{X}=0$).

Proof.

Let $(U,S)$ be a pair of random variables taking values $(u,s)\in V\times\Sigma$ with probability ${\mathsf{P}}(u,s)$. Let $\varphi:\Sigma\to\{L,I,H\}$ be the function that maps $s$ to $X$ if $s\in X$. Now, $U$—$S$—$\varphi(S)$ is a Markov chain, so that by the data processing inequality, ${\mathrm{I}}(U;S)\geq{\mathrm{I}}(U;\varphi(S))$. It is easily verified that ${\mathrm{I}}(U;S)={\mathsf{h}}({\mathbf{p}})-{\mathsf{H}}({\mathsf{P}})$ and ${\mathrm{I}}(U;\varphi(S))={\mathsf{h}}({\mathbf{\rho}})-{\mathsf{H}}({\mathsf{Q}})$. Thus,

[TABLE]

re-arranging which we obtain (12).

Equality holds in the data processing inequality above if and only if $U$—$\varphi(S)$—$S$ is also a Markov chain, i.e., $U$ and $S$ are conditionally independent given $\varphi(S)$. Now, check that $\Pr\{U=u,S=s\mid\varphi(S)=X\}$ equals ${\mathsf{P}}(u,s)/\rho_{X}$ if $s\in X$, and equals [math] otherwise. Hence, $\Pr\{U=u,S=s\mid\varphi(S)=X\}=\sum_{s\in X}{\mathsf{P}}(u,s)/\rho_{X}={\mathsf{Q}}(u,X)/\rho_{X}$. Finally, $\Pr\{S=s\mid\varphi(S)=X\}$ equals $p_{s}/\rho_{X}$ if $s\in X$, and equals [math] otherwise. Thus, the required conditional independence holds if and only if ${\mathsf{P}}(u,s)={\mathsf{Q}}(u,X)\,(p_{s}/\rho_{X})$ for all $s\in X$. ∎

We can now complete the proof of Theorem 4. Taking the supremum over ${\mathsf{P}}$ in (12), we obtain (by virtue of (4)) that

[TABLE]

We now argue that this is in fact an equality. Consider a {\mathsf{Q}}^{*}={\bigl{(}{\mathsf{Q}}^{*}(u,X)\bigr{)}}_{u,X} that achieves ${\mathsf{cap}}({\mathsf{S}}_{3;1,1},{\mathbf{\rho}})=\sup{\mathsf{H}}({\mathsf{Q}})$, the supremum being over Markov chains ${\mathsf{Q}}\in\Delta(G_{3;1,1})$ such that ${\mathbb{E}}_{{\mathsf{Q}}}({\mathcal{I}}_{\{L,I,H\}})={\mathbf{\rho}}$. Such a ${\mathsf{Q}}^{*}$ exists as ${\mathsf{Q}}\mapsto{\mathsf{H}}({\mathsf{Q}})$ is a continuous function being maximized over a compact set. Recall that any outgoing edge from $u$ labeled by $X$ in $G_{3;1,1}$ is replaced in $G_{q;a,b}$ by $|X|$ parallel edges labeled by the distinct symbols $s\in X$. For each such edge $(u,s)$, set ${\mathsf{P}}(u,s)=(p_{s}/\rho_{X}){\mathsf{Q}}^{*}(u,X)$. The resulting Markov chain ${\mathsf{P}}\in\Delta(G_{q;a,b})$ satisfies the conditions for equality in (12), from which it follows that equality holds in (13).

IV Discussion

Our computation of ${\mathsf{cap}}({\mathsf{S}}_{q;a,b},{\mathbf{p}})$ consists of the following steps.

Applying Theorem 4 to reduce the problem to that of computing ${\mathsf{cap}}({\mathsf{S}}_{3;1,1},{\mathbf{\rho}})$. 2. 2.

Expressing the computation of ${\mathsf{cap}}({\mathsf{S}}_{3;1,1},{\mathbf{\rho}})$ as the bivariate minimization problem (8) in the variables $(y,z)$. 3. 3.

Eliminating the implicit expression $\lambda(A(y,z))$ in (8) through a change of variables, resulting in the bivariate minimization problem (10) in the variables $(y,\lambda)$. 4. 4.

Taking partial derivatives with respect to $y$ and $\lambda$, resulting in two cubic bivariate polynomials in $y$ and $\lambda$. 5. 5.

Finding the common root of these polynomials.

While Step 1 is specific to the constrained system ${\mathsf{S}}_{q;a,b}$, the other steps might be applicable to other count-constrained systems (albeit with varying degrees of difficulty). For any constrained system ${\mathsf{S}}$ over an alphabet $\Sigma$, the number of variables in Step 2 will be $|\Sigma|-1$. As for Step 3, the explicit rational expression (9) for $z=z(y,\lambda)$ is attributed to the fact that the coefficients of the characteristic polynomial of $A(y,z)$ are linear terms in $z$. In general, this happens whenever there is a symbol $s\in\Sigma$ that has a “home state” in the graph presentation of ${\mathsf{S}}$, namely, all edges labeled by $s$ lead to the same vertex.

We mention that one could also compute ${\mathsf{cap}}({\mathsf{S}}_{q;a,b},{\mathbf{p}})$ based on Proposition 1 directly. Referring to the case ${\mathsf{cap}}({\mathsf{S}}_{3;1,1},{\mathbf{\rho}})$ and using the notation towards the end of Section III-B, such a computation would entail finding the nine edge probabilities of a Markov chain {\mathsf{Q}}={\bigl{(}{\mathsf{Q}}(u,X)\bigr{)}}_{u,X} (where $u\in V=\{1,2,3\}$ and $X\in\Sigma=\{L,I,H\}$) that maximizes ${\mathsf{H}}({\mathsf{Q}})$, subject to the following six linear constraints:

• •

${\mathsf{Q}}(2,H)=0$,

• •

the constraints (3) for any two vertices in $V$ (the third is dependent on these two), and—

• •

the three constraints obtained from ${\mathbb{E}}({\mathcal{I}}_{\Sigma})={\mathbf{\rho}}$ (these constraints imply that $\sum_{u,X}{\mathsf{Q}}(u,X)=1$).

We would then end up with three linearly independent variables to optimize over.

Acknowledgment

N. Kashyap and P. H. Siegel would like to acknowledge the support of the Indo–US Science & Technology Forum (IUSSTF), which funded in part the work reported here. This work was also supported in part by NSF Grant CCF-1619053, and by Grant 2015816 from the United-States–Israel Binational Science Foundation (BSF). Portions of this work were conducted while P. H. Siegel visited Technion in May 2013 and while R. M. Roth visited the Center for Memory and Recording Research (CMRR) at UC San Diego in summer 2018.

Appendix A Alternative Definition of Capacity

The definition of ${\mathsf{cap}}({\mathsf{S}}_{q;a,b},{\mathbf{p}})$ in [1] differs from ours in (1) and takes the form

[TABLE]

where ${\mathsf{S}}_{q;a,b}({\mathbf{p}})$ consists of all words ${\mathbf{w}}\in{\mathsf{S}}_{q;a,b}$ such that, for some prescribed $s_{0}\in L$, the number of occurrences of any other symbol $s\in\Sigma_{0}:=\Sigma\setminus\{s_{0}\}$ in ${\mathbf{w}}$ equals $\left\lfloor p_{s}|{\mathbf{w}}|\right\rfloor$ (and $s_{0}$ fills up the remaining positions).

Clearly, for any fixed $\varepsilon>0$ and sufficiently large $n$,

[TABLE]

and, therefore, (14) is bounded from above by ${\mathsf{cap}}({\mathsf{S}}_{q;a,b},{\mathbf{p}})$ as defined in (1).

We next turn to showing that (1) is also a lower bound on (14). We assume here that $p_{\min}:=\min_{s\in\Sigma_{0}}p_{s}>0$; the case where some entries of ${\mathbf{p}}$ are zero can then be argued by the continuity of ${\mathbf{p}}\mapsto{\mathsf{cap}}({\mathsf{S}}_{q;a,b},{\mathbf{p}})$ (at neighborhoods of vectors ${\mathbf{p}}$ with $p_{\min}=0$).666In particular, it can be verified that the second case in Theorem 3—that of $p_{L}=\frac{1}{2}$ and $p_{I}>0$—indeed converges to the first case therein.

We define a one-to-one mapping from ${\mathsf{S}}_{q;a,b}({\mathbf{p}},\varepsilon)\cap\Sigma^{n}$ into ${\mathsf{S}}_{q;a,b}({\mathbf{p}})\cap\Sigma^{n^{\prime}}$, where

[TABLE]

namely, $n^{\prime}$ is only “slightly larger” than $n$. The image of a word ${\mathbf{w}}\in{\mathsf{S}}_{q;a,b}({\mathbf{p}},\varepsilon)\cap\Sigma^{n}$ is a word ${\mathbf{w}}^{\prime}={\mathbf{w}}{\mathbf{w}}^{\prime\prime}$, where the suffix ${\mathbf{w}}^{\prime\prime}$ is determined as follows.

• •

If ${\mathbf{w}}$ ends with a symbol in $L\cup I$, then ${\mathbf{w}}^{\prime\prime}$ is any sequence of symbols from $L$, $I$, and $H$, in that order, so that the count of each symbol $s\in\Sigma_{0}$ in ${\mathbf{w}}^{\prime}$ reaches $\left\lfloor p_{s}n^{\prime}\right\rfloor$. The symbol $s_{0}$ then fills any vacant positions among those allocated to symbols of $L$.

• •

If ${\mathbf{w}}$ ends with a symbol in $H$, then we do the same except that the symbols from $L$ are filled last.

The count of each symbol $s\in\Sigma$ in ${\mathbf{w}}$ is bounded from above by

[TABLE]

In addition, from $\sum_{s\in\Sigma}p_{s}n^{\prime}=n^{\prime}$ we get

[TABLE]

and, since we assume that $p_{s_{0}}>0$,

[TABLE]

It follows from (15)–(16) that we should always be able to reach the targeted count, $\left\lfloor p_{s}n^{\prime}\right\rfloor$, in ${\mathbf{w}}^{\prime}$, for each symbol $s\in\Sigma_{0}$. Moreover, by (16), there will be at least one position in ${\mathbf{w}}^{\prime\prime}$ filled with $s_{0}$; so, when ${\mathbf{w}}$ ends with a symbol in $L$, that symbol will be followed in ${\mathbf{w}}^{\prime}$ by a symbol in $L$. Hence, the image ${\mathbf{w}}^{\prime}$ is indeed in ${\mathsf{S}}_{q;a,b}({\mathbf{p}})\cap\Sigma^{n^{\prime}}$.

We conclude that

[TABLE]

and taking logarithms, dividing by $n^{\prime}$, and then taking $n\rightarrow\infty$ yields that (14) is at least

[TABLE]

Finally, taking the limit $\varepsilon\rightarrow 0^{+}$ implies that our definition of ${\mathsf{cap}}({\mathsf{S}}_{q;a,b},{\mathbf{p}})$ in (1) is a lower bound on (14).

Appendix B Proof of $\partial z/\partial\lambda>0$

We show that if $z(y,\lambda)$ is as in (9), then $\partial z/\partial\lambda>0$ whenever $\lambda\geq y+1\geq 1$.

For $\lambda=y+1$ the numerator in (9) vanishes and therefore we have:

[TABLE]

When $\lambda>y+1$ we have $z>0$; we show that in this case, $\partial\lambda(A(y,z))/\partial z>0$. Given $(y,z)$ where $y\geq 0$ and $z>0$, let ${\mathbf{x}}={\mathbf{x}}(y,z)=(x_{1}\;x_{2}\;x_{3})^{T}$ be the unique (up to scaling) all-positive eigenvector that corresponds to $\lambda=\lambda(A(y,z))$:

[TABLE]

Let $\alpha$ be a positive real in $(0,1)$ that satisfies the inequality:

[TABLE]

It can be verified that for such an $\alpha$, the following inequality holds componentwise for any sufficiently small $\delta>0$:

[TABLE]

where $\varepsilon=\alpha\delta x_{2}/(y+1)$. By Theorem 5.4 in777The proof of the necessity part in that theorem does not require the entries of the matrix $A$ or the vector ${\mathbf{x}}$ to be integers. [4] it follows from (17) that $\lambda(A(y,z+\delta))\geq\lambda+\alpha\delta$. This, in turn, implies that $\alpha$ is a lower bound on $\partial\lambda(A(y,z))/\partial z$.

Appendix C Proof of Theorem 3

Consider $\rho_{I}>0$ first. Requiring $\partial g/\partial y$ to be [math] yields

[TABLE]

and requiring $\partial g/\partial\lambda$ to be [math] yields

[TABLE]

Rearranging terms in (18) in descending powers of $y$ results in:

[TABLE]

Also, computing $((1/\rho_{H})-1)\times(\ref{eq:dgdy})-(\rho_{I}/\rho_{H})\times(\ref{eq:dgdlambda})$, and recalling that $\rho_{L}=1-\rho_{H}-\rho_{I}$, we obtain (after rearranging terms in descending powers of $y$):

[TABLE]

Finally, subtracting (20) from (21) results in:

[TABLE]

The case $\rho_{L}={\frac{1}{2}}$ is somewhat special, as $y$ then disappears from (22) and we get $\lambda=2$. Plugging this value of $\lambda$ into (20) (or into (21)) yields the following equation for $y$:

[TABLE]

We solve the quadratic equation (23) for the positive root:

[TABLE]

where $\tau=\rho_{H}/\rho_{I}$. It is easy to check that $(y,2)\in{\mathcal{U}}$. This yields the case of $\rho_{L}={\frac{1}{2}}$ and $\rho_{I}>0$ in the statement of the theorem.

Assume hereafter that $\rho_{L}\neq{\frac{1}{2}}$. From (22) we can express $y$ in terms of $\lambda$:

[TABLE]

Substituting this value into (21) yields:

[TABLE]

This is a cubic equation in $\lambda$: $Z(\lambda)=0$, where the coefficients of $Z(x)=\sum_{i=0}^{3}x^{i}$ are given by

[TABLE]

Plugging $\rho_{I}=1-\rho_{L}-\rho_{H}$ into these expressions yields:

[TABLE]

Next, we find a root $\lambda$ of $Z(x)$ such that, along with $y$ as in (25), we get a point in ${\mathcal{U}}$. This point necessarily attains the infimum in (10). For the analysis, we will find it useful to re-write $Z(x)$ as

[TABLE]

We distinguish between two cases.

Case 1: $\rho_{L}<{\frac{1}{2}}$. It is easily seen that $Z(0)=(1-2\rho_{H})^{2}\geq 0$; moreover, if $Z(0)=0$ then [math] is a multiple root of $Z(x)$. From (27), it readily follows that

[TABLE]

We conclude that $Z(x)$ has three real roots: one in $(2,\infty)$, a second root in $[0,2)$, and a third root which is non-positive. Since $y$ in (25) has to be positive, it follows that $\lambda$ equals the unique root of $Z(x)$ which is in $(2,\infty)$.

Case 2: $\rho_{L}>{\frac{1}{2}}$. From (27), we also get that $Z(1/\rho_{L})=\left((\rho_{L}-\rho_{H})/\rho_{L}-(1-2\rho_{H})\right)^{2}=\rho_{H}^{2}\left(2-(1/\rho_{L})\right)^{2}>0$. Moreover, it is easily seen from (27) that $Z(x)$ is positive on the entire interval $(-\infty,1/\rho_{L}]$. Since $Z(2)<0$, it follows that $Z(x)$ has a root in $(1/\rho_{L},2)$ (and that root is the smallest real root of $Z(x)$). For such a root the value $y$ in (25) is positive.888In fact, $Z(x)$ must have a unique root in $(1/\rho_{L},2)$: otherwise, there would be two points, $(y^{(1)},z^{(2)})$ and $(y^{(2)},z^{(2)})$, with $z^{(1)}\neq z^{(2)}$, that would attain the infimum in (8). These points would correspond in a one-to-one manner to two distinct points, $(\xi_{I}^{(1)},\xi_{H}^{(2)})$ and $(\xi_{I}^{(2)},\xi_{H}^{(2)})$, that would attain the infimum of (LABEL:eq:R3p:1). Yet then, by the convexity of (LABEL:eq:R3p:1), all the points on the line that connects the latter two points would attain the infimum, thereby absurdly implying that $Z(x)$ has infinitely many roots.

In both cases, we have

[TABLE]

i.e., $(y,\lambda)\in{\mathcal{U}}$. This completes the analysis of the $\rho_{I}>0$ case.

To deal with the $\rho_{I}=0$ case, we set $y=0$. Recall that $g_{0}(\lambda):=g(0,\lambda)$. Setting $g_{0}^{\prime}(\lambda)=0$ yields (19) with $y=0$. If $\rho_{L}={\frac{1}{2}}$, then $\rho_{H}={\frac{1}{2}}$ as well, and (19) simplifies to $\lambda^{3}-2\lambda^{2}=0$, from which we obtain $\lambda=2$ as the only solution larger than $1$. This yields the case of $\rho_{L}={\frac{1}{2}}$ and $\rho_{I}=0$ in the theorem statement.

When $\rho_{L}\neq{\frac{1}{2}}$, then using the fact that $1-\rho_{H}=\rho_{L}$ (since $\rho_{I}=0$), we write the LHS of (19) (again with $y=0$) as

[TABLE]

On the other hand, using the fact that $1-2\rho_{L}=\rho_{H}-\rho_{L}=2\rho_{H}-1$, it can be verified that $Z(x)$ as in (26) is in fact equal to $(1-2\rho_{L})Z_{0}(x)$. Since $\rho_{L}\neq{\frac{1}{2}}$, $Z(x)$ and $Z_{0}(x)$ have the same roots, from which it follows that the theorem statements for the $\rho_{L}\neq{\frac{1}{2}}$ case apply when $\rho_{I}=0$ as well.