Rate-Distortion Region of a Gray-Wyner Model with Side Information

TL;DR

This paper provides a comprehensive single-letter characterization of the rate-distortion region for a Gray-Wyner model with side information at the decoders, considering correlated sources and multiple fidelity constraints.

Contribution

It introduces a full single-letter characterization of the rate-distortion region for the Gray-Wyner model with side information, including specializations to Heegard-Berger models.

Findings

Derived the rate-distortion region for the model with side information.

Analyzed the roles of common and private descriptions in coding.

Provided binary examples illustrating the theoretical results.

Abstract

In this work, we establish a full single-letter characterization of the rate-distortion region of an instance of the Gray-Wyner model with side information at the decoders. Specifically, in this model an encoder observes a pair of memoryless, arbitrarily correlated, sources $(S^n_1,S^n_2)$ and communicates with two receivers over an error-free rate-limited link of capacity $R_0$, as well as error-free rate-limited individual links of capacities $R_1$ to the first receiver and $R_2$ to the second receiver. Both receivers reproduce the source component $S^n_2$ losslessly; and Receiver $1$ also reproduces the source component $S^n_1$ lossily, to within some prescribed fidelity level $D_1$. Also, Receiver $1$ and Receiver $2$ are equipped respectively with memoryless side information sequences $Y^n_1$ and $Y^n_2$. Important in this setup, the side information sequences are arbitrarily correlated among them, and with the source pair $(S^n_1,S^n_2)$; and are not assumed to exhibit any particular ordering. Furthermore, by specializing the main result to two Heegard-Berger models with successive refinement and scalable coding, we shed light on the roles of the common and private descriptions that the encoder should produce and what they should carry optimally. We develop intuitions by analyzing the developed single-letter optimal rate-distortion regions of these models, and discuss some insightful binary examples.

Tables

Table 1. TABLE I: Auxiliary random variables associated with the subsets that appear in ( 33 ).

	${𝒯}_0$	${𝒯}_1$	${𝒯}_2$
${𝒜}_{{𝒯}_j}^{-}$	$\varnothing$	$\varnothing$	$U_1$
${𝒜}_{{𝒯}_j}^{\supset }$	$\varnothing$	$U_{12}$	$U_{12}$
${𝒜}_{{𝒯}_j}^{+}$	$\{U_1,U_2\}$	$\varnothing$	$\varnothing$
${𝒜}_{{𝒯}_j}^{†}$	$\varnothing$	$\varnothing$	$\varnothing$
${𝒜}_{{𝒯}_j,1}^{‡}$	$\varnothing$	$\varnothing$	$\varnothing$
${𝒜}_{{𝒯}_j,2}^{‡}$	$\varnothing$	$\varnothing$	$\varnothing$

Equations

$$\begin{array}[]{rcl}d_{1}\ :\ \mathcal{S}_{1}\times\hat{\mathcal{S}}_{1}&\rightarrow&\mathbb{R}_{+}\\ (s_{i},\hat{s}_{i})&\rightarrow&d_{1}(s_{i},\hat{s}_{i})\ .\end{array}$$

$$\begin{array}[]{rcl}f_{j}\ :\ \mathcal{S}^{n}_{1}\times\mathcal{S}_{2}^{n}&\rightarrow&\mathcal{W}_{j}\\ (S_{1}^{n},S_{2}^{n})&\rightarrow&W_{j}=f_{j}(S_{1}^{n},S_{2}^{n})\ .\end{array}$$

$$\begin{array}[]{rcl}g_{1}\ :\ \mathcal{W}_{0}\times\mathcal{W}_{1}\times\mathcal{Y}^{n}_{1}&\rightarrow&\hat{\mathcal{S}}_{2}^{n}\times\hat{\mathcal{S}}_{1}^{n}\\ (W_{0},W_{1},Y_{1}^{n})&\rightarrow&(\hat{S}_{2,1}^{n},\hat{S}_{1}^{n})=g_{1}(W_{0},W_{1},Y_{1}^{n})\ ,\end{array}$$

$$\begin{array}[]{rcl}g_{2}\ :\ \mathcal{W}_{0}\times\mathcal{W}_{2}\times\mathcal{Y}^{n}_{2}&\rightarrow&\hat{\mathcal{S}}_{2}^{n}\\ (W_{0},W_{2},Y_{2}^{n})&\rightarrow&\hat{S}_{2,2}^{n}=g_{2}(W_{0},W_{2},Y_{2}^{n})\ .\end{array}$$

$$\mathbb{E}\left(d_1^{(n)}(S_1^n,{\hat{S}}_1^n)\right)\triangleq \mathbb{E}\frac{1}{n}\sum _{i=1}^n{d}_1(S_{1,i},{\hat{S}}_{1,i}).$$

$$P^{(n)}_{e}\triangleq\mathds{P}\bigl{(}\hat{S}_{2,1}^{n}\neq S_{2}^{n}\ \text{or}\ \hat{S}_{2,2}^{n}\neq S_{2}^{n}\bigr{)}\ .$$

$$\underset{n\to \infty }{\mathrm{limsup}}{P}_e^{(n)}$$

$$\underset{n\to \infty }{\mathrm{limsup}}\mathbb{E}\left(d_1^{(n)}(S_1^n,{\hat{S}}_1^n)\right)$$

$$\underset{n\to \infty }{\mathrm{liminf}}\frac{1}{n}{\log }_2(M_{j,n})$$

$$\mathcal{RD}\triangleq\cup\bigl{\{}(R_{0},R_{1},R_{2},D_{1}):(R_{0},R_{1},R_{2})\text{ is $D_{1}$-achievable}\bigr{\}}\ .$$

$$R_0+R_1$$

$$R_0+R_2$$

$$R_0+R_1+R_2$$

$$(Y_1,Y_2)-\circ --(S_1,S_2)-\circ --(U_0,U_1)$$

$$\text{mathds}E{d}_1(S_1,{\hat{S}}_1)\le D_1.$$

$$R_0$$

$$R_0+R_1$$

$$(U_0,U_1)-\circ --(S_1,S_2)-\circ --(Y_1,Y_2)$$

$$\text{mathds}E{d}_1(S_1,{\hat{S}}_1)\le D_1.$$

$$R_0+R_1\ge \min \left\{I(U_0{S}_2;S_1{S}_2|Y_1),I(U_0{S}_2;S_1{S}_2|Y_2)\right\}+I(U_1;S_1|U_0{S}_2{Y}_1)$$

$$R_0$$

$$R_0+R_1$$

$$R_0$$

$$R_0+R_1$$

$$R_0$$

$$R_0+R_1$$

$$=H(S_1{S}_2|Y_1)$$

$$=2$$

$$R_0$$

$$=2-H(X_1|X_2{X}_3{X}_4{U}_0)$$

$$\ge 2-H(X_1|X_2{X}_3{X}_4)$$

$$=1$$

$$R_0+R_1$$

$$=2+H(X_2{X}_3|X_1{X}_4{U}_0)-H(X_1|X_2{X}_3{X}_4{U}_0)$$

$$=2+\left[H(X_2{X}_3|X_4{U}_0)-H(X_1|X_4{U}_0)\right]$$

$$R_0+R_1$$

Full text

I Introduction

The Gray-Wyner source coding problem was originally formulated, and solved, by Gray and Wyner in [1]. In their original setting, referred therein to as a ”simple network”, a pair of arbitrarily correlated memoryless sources $(S^{n}_{1},S^{n}_{2})$ is to be encoded and transmitted to two receivers that are connected to the encoder each through a common error-free rate-limited link as well as a private error-free rate-limited link. Because the channels are rate-limited, the encoder produces a compressed bit string $W_{0}$ of rate $R_{0}$ that it transmits over the common link, and two compressed bit strings, $W_{1}$ of rate $R_{1}$ and $W_{2}$ of rate $R_{2}$, that it transmits respectively over the private link to first receiver and the private link to the second receiver. The first receiver uses the bit strings $W_{0}$ and $W_{1}$ to reproduce an estimate, or approximation, $\hat{S}^{n}_{1}$ of the source component $S^{n}_{1}$ to within some prescribed average fidelity level $D_{1}$, for some distortion measure $d_{1}(\cdot,\cdot)$. Similarly, the second receiver uses the bit strings $W_{0}$ and $W_{2}$ to reproduce an estimate $\hat{S}^{n}_{2}$ of the source component $S^{n}_{2}$ to within some prescribed average fidelity level $D_{2}$, for some, possibly different, distortion measure $d_{2}(\cdot,\cdot)$. In [1], Gray and Wyner characterized the optimal tradeoff among achievable rate triples $(R_{0},R_{1},R_{2})$ and distortion pair $(D_{1},D_{2})$.

Figure 1 shows a generalization of Gray-Wyner’s original model in which the receivers also observe, or measure, correlated memoryless side information sequences, $Y^{n}_{1}$ at Receiver $1$ and $Y^{n}_{2}$ at Receiver $2$. Some special cases of the Gray-Wyner’s model with side information of Figure 1 have been solved (see the ”Related Work” section below). However, in its most general form, i.e., when the side information sequences are arbitrarily correlated among them and with the sources, this problem has so-far eluded single-letter characterization of the optimal rate-distortion region, which is then still to be found. In fact, the problem appears somewhat hopeless as the rate-distortion function of a specific case of it, the well known Heegard-Berger problem [2], which is obtained by setting $R_{1}=R_{2}=0$ in Figure 1, has itself eluded the information theory for, now, more than three decades.

In this paper, we study an instance of the Gray-Wyner’s model with side information of Figure 1 in which both receivers want to reproduce the source component $S^{n}_{2}$ losslessly; and Receiver $1$ also wants to reproduce the source component $S^{n}_{1}$ lossily, to within some prescribed fidelity level $D_{1}$. The model is shown in Figure 2; and we refer to it as “Gray-Wyner model with side information and degraded reconstruction sets”. It is important to note that, while the reconstruction sets are assumed to be degraded, no specific ordering is imposed on the side information sequences, which then can be arbitrarily correlated among them and with the sources $(S^{n}_{1},S^{n}_{2})$.

Typical to in similar problems, the encoder produces a common description of the sources pair $(S^{n}_{1},S^{n}_{2})$ that is intended to be recovered by both receivers, as well as individual or private descriptions of $(S^{n}_{1},S^{n}_{2})$ that are destined to be recovered each by a distinct receiver. Because the side information sequences do not exhibit any specific ordering, it is not clear a-priori what information these descriptions should carry optimally. Also, it is not clear how these descriptions should be transmitted using the available common and dedicated links.

Instrumental in the investigation of the general model of Figure 2, and helpful for the reader to grasp the intuitions gradually, we will also study the two important underlying Heegard-Berger special models that are shown in Figure 3. In both models, only one of the two refinement individual links has non-zero rate. In the model of Figure 3a, the receiver that accesses the additional rate-limited link (i.e., Receiver $1$) is also required to reproduce a lossy estimate of the source component $S^{n}_{1}$, in addition to the source component $S^{n}_{2}$ which is to be reproduced losslessly by both receivers. We will refer to this model as a “Heegard-Berger problem with successive refinement”, in reference to that if a user gets only a coarse description of the sources it should only reproduce the source component $S^{n}_{2}$ (losslessly); and if, at a later stage, a description of the source component $S^{n}_{1}$ as well is needed, the encoder sends a secondary string of compressed bits at rate $R_{1}$ to that user. Reminiscent of successive refinement source coding, which is a special case of the more general multiple description coding, this model may be appropriate to model applications in which descriptions of only some components (e.g., $S^{n}_{2}$) of the source suffices at the first use of the data; and descriptions of the remaining components (e.g., $S^{n}_{1}$) are needed only at a later stage.

The model of Figure 3b has the individual rate-limited link connected to the receiver that is required to reproduce only the source component $S^{n}_{2}$. From an application viewpoint, the reader may find it appropriate to think about the second receiver here as having a “lower quality” side information; and the private link is introduced precisely so that the user with the better side information and/or capability be not constrained by the communication to the user with bad and/or less good side information or capability. We will refer to this model as a “Heegard-Berger problem with scalable coding”, reusing a term that was coined in [3] for a similar scenario, and in reference to that a user may have such strong capability (e.g., thanks to a strong side information) that only a minimal amount of information from the encoder suffices; or, conversely, barely any capability (e.g., due to weak side information) that the encoder should provide almost every information to it in order to satisfy a fidelity criterion. However, unlike [3], we do no make any assumption on the ordering of the side information sequences in our analysis; and the results that will follow for this model hold in general as we already mentioned.

I-A Main Contributions

The main result of this paper is a single-letter characterization of the optimal rate-distortion region of the Gray-Wyner model with side information and degraded reconstruction sets of Figure 2. To this end, in particular, we derive a converse proof that is tailored specifically for the model with degraded reconstruction sets that we study here. For the proof of the direct part, we develop a coding scheme that is very similar to one developed in the context of coding for broadcast channels with feedback in [4], but with an appropriate choice of the variables which we specify here. The specification of the main result to the Heegard-Berger models with successive refinement and scalable coding of Figure 3 sheds important light on the roles of the common and private descriptions and what they should carry optimally. We develop intuitions by analysing the established single-letter optimal rate-distortion regions of these models, and discuss some insightful binary examples.

I-B Related Work

In [4], Shayevitz and Wigger study a two-receiver discrete memoryless broadcast channel with feedback. They develop an efficient coding scheme which treats the feedback signal as a source that has to be conveyed lossily to the receivers in order to refine their messages’ estimates, through a block Markov coding scheme. In doing so, the users’ channel outputs are regarded as side information sequences; and so the scheme clearly connects with the Gray-Wyner model with side information of Figure 1 - as is also clearly explicit in  [4]. The Gray-Wyner model with side information for which Shayevitz and Wigger’s develop a (source) coding scheme, as part of their study of the broadcast channel with feedback, assumes general, possibly distinct, distortion measures at the receivers (i.e., not necessarily nested) and side information sequences that are arbitrarily correlated among them and with the source. However, its optimality is still to be shown (if it were to hold). As we already mentioned, in this paper we show that when specialized to the model with degraded reconstruction sets of Figure 2 that we study here, Shayevitz and Wigger’s coding scheme for the Gray-Wyner model with side information of [4] yields a rate-distortion region that meets the converse result that we here establish; and so is optimal.

The Gray-Wyner model with side information generalizes another long standing open source coding problem, the famous Heegard-Berger problem [5]. Full single-letter characterization of the optimal rate-distortion function of the Heegard-Berger problem is known only in few specific cases, the most important of which are the cases of i) stochastically degraded side information sequences [5] (see also [6]), ii) Sgarro’s result [7] on the corresponding lossless problem, iii) Gaussian sources with quadratic distortion measure [8, 3], iv) some instances of conditionally less-noisy side information sequences [9] and v) the recently solved HB model with general side information sequences and degraded reconstruction sets [10], i.e., the model of Figure 2 with $R_{1}=R_{2}=0$ — in the lossless case, a few other optimal results were shown, such as for the so-called complementary delivery [11]. A lower bound for general instances of the rate distortion problem with side information at multiple decoders, that is inspired by a linear-programming lower bound for index coding, has been developed recently by Unal and Wagner in [12].

Successive refinement of information was investigated by Equitz et al. in [13] wherein the description of the source is successively refined to a collection of receivers which are required to reconstruct the source with increasing quality levels. Extensions of successive refinement to cases in which the receivers observe some side information sequences was first investigated by Steinberg et al. in [14] who establish the optimal rate-distortion region under the assumption that the receiver that observes the refinement link, say receiver $1$, observes also a better side information sequence than the opposite user, i.e. the following Markov chain $S-\!\!\!\!\minuso\!\!\!\!-Y_{1}-\!\!\!\!\minuso\!\!\!\!-Y_{2}$ holds. Tian et al. give in [8] an equivalent formulation of the result of [14] and extend it to the N-stage successive refinement setting. In [3], Tian et al. investigate another setting, for which they coined the term “side information scalabale coding”, in which it is rather the receiver that accesses the refinement link, say receiver $2$, which observes the less good side information sequence, i.e. $S-\!\!\!\!\minuso\!\!\!\!-Y_{1}-\!\!\!\!\minuso\!\!\!\!-Y_{2}$. Balancing refinement quality and side information asymmetry for such a side-information scalable source coding problem allows authors in [3] to derive the rate-distortion region in the degraded side information case. The previous results on successive refinement in the presence of side information, which were generalized by Timo et al. in [15], all assume, however, a specific structure in the side information sequences.

I-C Outline and Notation

An outline of the remainder of this paper is as follows. Section II describes formally the Gray-Wyner model with side information and degraded reconstruction sets of Figure 2 that we study in this paper. This section also contains some formal definitions for the Heegard-Berger type models of Figure 3. Section III contains the main result of this paper, a full single-letter characterization of the rate-distortion region of the model of Figure 2, together with some useful discussions and connections. A formal proof of the direct and converse parts of this result appear in Section VI. In Section IV and Section V, we specialize the result respectively to the Heegard-Berger model with successive refinement of Figure 3a and the Heegard-Berger model with scalable coding of Figure 3b. These sections also contains insightful discussions and binary examples.

Throughout the paper we use the following notations. The term p.m.f. stands for probability mass function. Upper case letters are used to denote random variables, e.g., $X$; lower case letters are used to denote realizations of random variables, e.g., $x$; and calligraphic letters designate alphabets, i.e., $\mathcal{X}$. Vectors of length $n$ are denoted by $X^{n}=(X_{1},\dots,X_{n})$, and $X_{i}^{j}$ is used to denote the sequence $(X_{i},\dots,X_{j})$. The probability distribution of a random variable $X$ is denoted by $P_{X}(x)\triangleq\mathds{P}(X=x)$. Sometimes, for convenience, we write it as $P_{X}$. We use the notation $\mathbb{E}_{X}[\cdot]$ to denote the expectation of random variable $X$. A probability distribution of a random variable $Y$ given $X$ is denoted by $P_{Y|X}$. The set of probability distributions defined on an alphabet $\mathcal{X}$ is denoted by $\mathcal{P}(\mathcal{X})$. The cardinality of a set $\mathcal{X}$ is denoted by $\|\mathcal{X}\|$. For random variables $X$, $Y$ and $Z$, the notation $X-\!\!\!\!\minuso\!\!\!\!-Y-\!\!\!\!\minuso\!\!\!\!-Z$ indicates that $X$, $Y$ and $Z$, in this order, form a Markov Chain, i.e., $P_{XYZ}(x,y,z)=P_{Y}(y)P_{X|Y}(x|y)P_{Z|Y}(z|y)$. The set $\mathcal{T}^{(n)}_{[X]}$ denotes the set of sequences strongly typical with respect to the probability distribution $P_{X}$ and the set $\mathcal{T}^{(n)}_{[X|y^{n}]}$ denotes the set of sequences $x^{n}$ jointly typical with $y^{n}$ with respect to the joint p.m.f. $P_{XY}$. Throughout this paper, we use $h_{2}(\alpha)$ to denote the entropy of a Bernoulli $(\alpha)$ source, i.e., $h_{2}(\alpha)=-\alpha\log(\alpha)-(1-\alpha)\log(1-\alpha)$. Also, the indicator function is denoted by $\mathds{1}(\cdot)$. For real-valued scalars $a$ and $b$, with $a\leq b$, the notation $[a,b]$ means the set of real numbers that are larger or equal than $a$ and smaller or equal $b$. For integers $i\leq j$, $[i:j]$ denotes the set of integers comprised between $i$ and $j$, i.e., $[i:j]=\{i,i+1,\ldots,j\}$. Finally, throughout the paper, logarithms are taken to base $2$.

II Problem Setup and Formal Definitions

Consider the Gray-Wyner source coding model with side information and degraded reconstruction sets shown in Figure 2. Let $(\mathcal{S}_{1}\times\mathcal{S}_{2}\times\mathcal{Y}_{1}\times\mathcal{Y}_{2},P_{S_{1},S_{2},Y_{1},Y_{2}})$ be a discrete memoryless vector source with generic variables $S_{1}$, $S_{2}$, $Y_{1}$ and $Y_{2}$. Also, let $\hat{\mathcal{S}}_{1}$ be a reconstruction alphabet and, $d_{1}$ a distortion measure defined as:

[TABLE]

Definition 1**.**

*An $(n,M_{0,n},M_{1,n},M_{2,n},D_{1})$ code for the Gray-Wyner source coding model with side information and degraded reconstruction sets of Figure 2 consists of:

• Three sets of messages $\mathcal{W}_{0}\triangleq[1:M_{0,n}]$, $\mathcal{W}_{1}\triangleq[1:M_{1,n}]$, and $\mathcal{W}_{2}\triangleq[1:M_{2,n}]$.

• Three encoding functions, $f_{0}$, $f_{1}$ and $f_{2}$ defined, for $j\in\{0,1,2\}$ as*

[TABLE]

- Two decoding functions $g_{1}$ and $g_{2}$, one at each user:

[TABLE]

and

[TABLE]

The expected distortion of this code is given by

[TABLE]

The probability of error is defined as

[TABLE]

∎

Definition 2**.**

A rate triple $(R_{0},R_{1},R_{2})$ is said to be $D_{1}$-achievable for the Gray-Wyner source coding model with side information and degraded reconstruction sets of Figure 2 if there exists a sequence of $(n,M_{0,n},M_{1,n},M_{2,n},D_{1})$ codes such that:

[TABLE]

The rate-distortion region $\mathcal{RD}$ of this problem is defined as the union of all rate-distortion quadruples $(R_{0},R_{1},R_{2},D_{1})$ such that $(R_{0},R_{1},R_{2})$ is $D_{1}$-achievable, i.e,

[TABLE]

∎

As we already mentioned, we shall also study the special case Heegard-Berger type models shown in Figure 3. The formal definitions for these models are similar to the above, and we omit them here for brevity.

III Gray-Wyner Model with Side Information and Degraded Reconstruction Sets

We establish a single-letter characterization of the optimal rate-distortion region $\mathcal{RD}$ of the Gray-Wyner model with side information and degraded reconstructions sets shown in Figure 2. The following theorem states the result.

Theorem 1**.**

The rate-distortion region $\mathcal{RD}$ of the Gray-Wyner problem with side information and degraded reconstruction set of Figure 2 is given by the sets of all rate-distortion quadruples $(R_{0},R_{1},R_{2},D_{1})$ satisfying:

[TABLE]

for some product pmf $P_{U_{0}U_{1}S_{1}S_{2}Y_{1}Y_{2}}$, such that:

1) the following Markov chain is valid:

[TABLE]

2) and there exists a function $\phi:\mathcal{Y}_{1}\times\mathcal{U}_{0}\times\mathcal{U}_{1}\times\mathcal{S}_{2}\rightarrow\hat{\mathcal{S}}_{1}$ such that:

[TABLE]

Proof: An outline of the proof of achievability of Theorem 1 will follow. The detailed scheme and its analysis, as well as the proof of the converse part, appear in Section VI.

Outline of Proof of Achievability: The encoder produces a common description of $(S^{n}_{1},S^{n}_{2})$ that is intended to be recovered by both receivers, and an individual description that is intended to be recovered only by Receiver $1$. The common description is $V^{n}_{0}=(U^{n}_{0},S^{n}_{2})$; and is designed so as to involve all of $S^{n}_{2}$, which both receivers are required to reproduce lossessly, but also all or part of $S^{n}_{1}$, depending on the desired fidelity level $D_{1}$. Since we make no assumption on the side information sequences, this is meant to account for possibly unbalanced side information pairs $(Y^{n}_{1},Y^{n}_{2})$, in a manner that is similar to [16, 10]. The message that carries the common description is obtained at the encoder through the technique of double-binning developed by Tian and Diggavi in [3], and then used by Shayevitz and Wigger [4, Theorem 2] for a Gray-Wyner model with side information. In particular, similar to the coding scheme of  [4, Theorem 2], the double-binning is performed in two ways, one that is relevant for Receiver $1$ and one that is relevant for Receiver $2$. Specifically, the codebook of the common description is composed of codewords $v^{n}_{0}$ that are drawn randomly and independently according to the product law of $P_{V_{0}}$; and is partitioned uniformly into $2^{n\tilde{R}_{0,0}}$ superbins, indexed with $\tilde{w}_{0,0}\in[1:2^{n\tilde{R}_{0,0}}]$. The codewords of each superbin of this codebook are partitioned in two distinct ways. In the first partition, they are assigned randomly and independently to $2^{n\tilde{R}_{0,1}}$ subbins indexed with $\tilde{w}_{0,1}\in[1:2^{n\tilde{R}_{0,1}}]$, according to a uniform pmf over $[1:2^{n\tilde{R}_{0,1}}]$. Similarly, in the second partition, they are assigned randomly and independently to $2^{n\tilde{R}_{0,2}}$ subbins indexed with $\tilde{w}_{0,2}\in[1:2^{n\tilde{R}_{0,2}}]$, according to a uniform pmf over $[1:2^{n\tilde{R}_{0,2}}]$. The codebook of the individual description is composed of codewords $u^{n}_{1}$ that are drawn randomly and independently according to the product law of $P_{U_{1}|V_{0}}$. This codebook is partitioned similarly uniformly into $2^{n\tilde{R}_{1,0}}$ superbins indexed with $\tilde{w}_{1,0}\in[1:2^{n\tilde{R}_{1,0}}]$, each containing $2^{n\tilde{R}_{1,1}}$ subbins indexed with $\tilde{w}_{1,1}\in[1:2^{n\tilde{R}_{1,1}}]$ codewords $u^{n}_{1}$.

Upon observing a typical pair $(S^{n}_{1},S^{n}_{2})=(s^{n}_{1},s^{n}_{2})$, the encoder finds a pair of codewords $(v^{n}_{0},u^{n}_{1})$ that is jointly typical with $(s^{n}_{1},s^{n}_{2})$. Let $\tilde{w}_{0,0}$, $\tilde{w}_{0,1}$ and $\tilde{w}_{0,2}$ denote respectively the indices of the superbin, subbin of the first partition and subbin of the second partition of the codebook of the common description, in which lies the found $v^{n}_{0}$. Similarly, let $\tilde{w}_{1,0}$ and $\tilde{w}_{1,1}$ denote respectively the indices of the superbin and subbin of the codebook of the individual description in which lies the found $u^{n}_{1}$. The encoder sets the common message $W_{0}$ as $W_{0}=(\tilde{w}_{0,0},\tilde{w}_{1,0})$ and sends it over the error-free rate-limited common link of capacity $R_{0}$. Also, it sets the individual message $W_{1}$ as $W_{1}=(\tilde{w}_{0,1},\tilde{w}_{1,1})$ and sends it the error-free rate-limited link to Receiver $1$ of capacity $R_{1}$; and the individual message $W_{2}$ as $W_{2}=\tilde{w}_{0,2}$ and sends it the error-free rate-limited link to Receiver $2$ of capacity $R_{2}$. For the decoding, Receiver $2$ utilizes the second partition of the codebook of the common description; and looks in the subbin of index $\tilde{w}_{0,2}$ of the superbin of index $\tilde{w}_{0,0}$ for a unique $v^{n}_{0}$ that is jointly typical with its side information $y^{n}_{2}$. Receiver $1$ decodes $v^{n}_{0}$ similarly, utilizing the first partition of the codebook of the common description and its side information $y^{n}_{1}$. It also utilizes the codebook of the individual description; and looks in the subbin of index $\tilde{w}_{1,1}$ of the superbin of index $\tilde{w}_{1,1}$ for a unique $u^{n}_{1}$ that is jointly typical with the pair $(y^{n}_{1},v^{n}_{0})$. In the formal proof in Section IV, we argue that with an appropropriate choice of the communication rates $\tilde{R}_{0,0}$, $\tilde{R}_{0,1}$, $\tilde{R}_{0,2}$, $\tilde{R}_{1,0}$ and $\tilde{R}_{1,1}$, as well as the sizes of the subbins, this scheme achieves the rate-distortion region of Theorem 1. ∎

A few remarks that connect Theorem 1 to known results on related models are in order.

Remark 1**.**

In the special case in which $R_{1}=R_{2}=0$, the Gray-Wyner model with side information and degraded reconstruction sets of Figure 2 reduces to a Heegard-Berger problem with arbitrary side information sequences and degraded reconstruction sets, a model that was studied, and solved, recently in the authors’ own recent work [10]. Theorem 1 can then be seen as a generalization of [10, Theorem1] to the case in which the encoder is connected to the receivers also through error-free rate-limited private links of capacity $R_{1}$ and $R_{2}$ respectively. ∎

Remark 2**.**

In [17], Timo et al. study the Gray-Wyner source coding model with side information of Figure 1. They establish the rate-region of this model in the specific case in which the side information sequence $Y^{n}_{2}$ is a degraded version of $Y^{n}_{1}$, i.e., $(S_{1},S_{2})-\!\!\!\!\minuso\!\!\!\!-Y_{1}-\!\!\!\!\minuso\!\!\!\!-Y_{2}$ is a Markov chain, and both receivers reproduce the component $S^{n}_{2}$ and Receiver $1$ also reproduces the component $S^{n}_{1}$, all in a lossless manner. The result of Theorem 1 generalizes that of [17, Theorem 5] to the case of side information sequences that are arbitrarily correlated among them and with the source pair $(S_{1},S_{2})$ and lossy reconstruction of $S_{1}$. In  [17], Timo et al. also investigate, and solve, a few other special cases of the model, such as those of single source $S_{1}=S_{2}$ [17, Theorem 4] and complementary delivery $(Y_{1},Y_{2})=(S_{2},S_{1})$ [17, Theorem 6]. The results of [17, Theorem 4] and  [17, Theorem 6] can be recovered from Theorem 1 as special cases of it. Theorem 1 also generalizes [17, Theorem 6] to the case of lossy reproduction of the component $S^{n}_{1}$. ∎

IV The Heegard-Berger Problem with Successive Refinement

An important special case of the Gray-Wyner source coding model with side information and degraded reconstruction sets of Figure 2 is the case in which $R_{2}=0$. The resulting model, which is of successive-refinement type for a Heegard-Berger problem as we already mentioned, is shown in Figure 3a. In this section, we specialize the result of the previous section to this setting and discuss a binary example. It is hoped that the below analysis and discussions shed more light on the roles of the common and individual descriptions that are produced by the encoder in this case, as well as on the way they are transmitted optimally over the available links.

IV-A Rate-Distortion Region

The following theorem states the optimal rate-distortion region of the Heegard-Berger problem with successive refinement of Figure 3a.

Theorem 2**.**

The rate-distortion region of the Heegard-Berger problem with successive refinement of Figure 3a is given by the set of rate-distortion triples $(R_{0},R_{1},D_{1})$ satisfying:

[TABLE]

for some product pmf $P_{U_{0}U_{1}S_{1}S_{2}Y_{1}Y_{2}}$, such that:

1) the following Markov chain is valid:

[TABLE]

2) and there exists a function $\phi:\mathcal{Y}_{1}\times\mathcal{U}_{0}\times\mathcal{U}_{1}\times\mathcal{S}_{2}\rightarrow\hat{\mathcal{S}}_{1}$ such that:

[TABLE]

Proof: The proof of Theorem 2 follows from that of Theorem 1 by setting $R_{2}=0$ therein.

Remark 3**.**

Recall the coding scheme of Theorem 1. If $R_{2}=0$, the second partition of the codebook of the common description, which is relevant for Receiver $2$, becomes degenerate since, in this case, all the codewords $v^{n}_{0}$ of a superbin $\mathcal{B}_{00}(\tilde{w}_{0,0})$ are assigned to a single subbin. Correspondingly, the common message that the encoder sends over the common link carries only the index $\tilde{w}_{0,0}$ of the superbin $\mathcal{B}_{00}(\tilde{w}_{0,0})$ of the codebook of the common description in which lies the found typical $v^{n}_{0}=(s^{n}_{2},u^{n}_{0})$, in addition to the index $\tilde{w}_{1,0}$ of the subbin $\mathcal{B}_{10}(\tilde{w}_{1,0})$ of the codebook of the individual description in which lies the found typical $u^{n}_{1}$. The constraint (14a) on the common rate $R_{0}$ is in accordance with that Receiver $2$ utilizes only the index $\tilde{w}_{0,0}$ in the decoding. Furthermore, note that the constraints (14b) and  (14c) on the sum-rate $(R_{0}+R_{1})$ can be combined as

[TABLE]

which readers who are familiar with Heegard-Berger type coding may recognize more easily (see, e.g.,  [2, Theorem 2, p. 733]).

Remark 4**.**

As we already mentioned, the result of Theorem 2 holds for side information sequences that are arbitrarily correlated among them and with the sources. In the specific case in which the user who gets the refinement rate-limited link also has the ”better-quality” side information, in the sense that $(S_{1},S_{2})-\!\!\!\!\minuso\!\!\!\!-Y_{1}-\!\!\!\!\minuso\!\!\!\!-Y_{2}$ forms a Markov chain, the rate-distortion region of Theorem 2 reduces to the set of all rate-distortion triples $(R_{0},R_{1},D_{1})$ that satisfy

[TABLE]

for some joint measure $P_{U_{0}U_{1}S_{1}S_{2}Y_{1}Y_{2}}$ for which (15) and (16) hold. This result can also be obtained from previous works on successive refinement for the Wyner-Ziv source coding problem by Steinberg and Merhav  [14, Theorem 1] and Tian and Diggavi [8, Theorem 1]. The results of [14, Theorem 1] and [8, Theorem 1] hold for possibly distinct, i.e., not necessarily nested, distortion measures at the receivers; but they require the aforementioned Markov chain condition which is pivotal for their proofs. Thus, for the considered degraded reconstruction sets setting, Theorem 2 can be seen as generalizing [14, Theorem 1] and [8, Theorem 1] to the case in which the side information sequences are arbitrarily correlated among them and with the sources $(S_{1},S_{2})$, i.e., do not exhibit any ordering. ∎

Remark 5**.**

In the case in which it is the user who gets only the common rate-limited link that has the ”better-quality” side information, in the sense that $(S_{1},S_{2})-\!\!\!\!\minuso\!\!\!\!-Y_{2}-\!\!\!\!\minuso\!\!\!\!-Y_{1}$ forms a Markov chain, the rate distortion region of Theorem 2 reduces to the set of all rate-distortion triples $(R_{0},R_{1},D_{1})$ that satisfy

[TABLE]

for some joint measure $P_{U_{0}U_{1}S_{1}S_{2}Y_{1}Y_{2}}$ for which (15) and (16) hold. This result can also be conveyed from [3]. Specifically, in  [3] Tian and Diggavi study a therein referred to as ”side-information scalable” source coding setup where the side informations are degraded, and the encoder produces two descriptions such that the receiver with the better-quality side information (Receiver $2$ if $(S_{1},S_{2})-\!\!\!\!\minuso\!\!\!\!-Y_{2}-\!\!\!\!\minuso\!\!\!\!-Y_{1}$ is a Markov chain) uses only the first description to reconstruct its source while the receiver with the low-quality side information (Receiver $1$ if $(S_{1},S_{2})-\!\!\!\!\minuso\!\!\!\!-Y_{2}-\!\!\!\!\minuso\!\!\!\!-Y_{1}$ is a Markov chain) uses the two descriptions in order to reconstruct its source. They establish inner and outer bounds on the rate-distortion region of the model, which coincide when either one of the decoders requires a lossless reconstruction or when the distortion measures are degraded and deterministic. Similar to in the previous remark, Theorem 2 can be seen as generalizing the aforementioned results of [3] to the case in which the side information sequences are arbitrarily correlated among them and with the sources $(S_{1},S_{2})$.∎

IV-B Binary Example

Let $X_{1}$, $X_{2}$, $X_{3}$ and $X_{4}$ be four independent $\text{Ber}(1/2)$ random variables. Let the sources be $S_{1}\triangleq(X_{1},X_{2},X_{3})$ and $S_{2}\triangleq X_{4}$. Now, consider the Heegard-Berger model with successive refinement shown in Figure 4. The first user, which gets both the common and individual links, observes the side information $Y_{1}=(X_{1},X_{4})$ and wants to reproduce the pair $(S_{1},S_{2})$ losslessly. The second user gets only the common link, has side information $Y_{2}=(X_{2},X_{3})$ and wants to reproduce only the component $S_{2}$, losslessly.

It is easy to check that the side information at the decoders do not exhibit any degradedness ordering, in the sense that none of the Markov chain conditions of Remark 4 and Remark 5 hold. The following claim provides the rate-region of this binary example.

Claim 1**.**

The rate region of the binary Heegard-Berger example with successive refinement of Figure 4 is given by the set of rate pairs $(R_{0},R_{1})$ that satisfy

[TABLE]

Proof: The proof of Claim 1 follows easily by specializing, and computing, the result of Theorem 2 for the example at hand, as follows. First note that for $D_{1}=0$ and the variables $(S_{1},S_{2},Y_{1},Y_{2})$ chosen as in the example, the inequality (14b) gives

[TABLE]

where the first equality follows since $H(S_{1}|S_{2}Y_{1}U_{0}U_{1})=0$ for all random variables $U_{0}$ and $U_{1}$ that satisfy (15) and (16), by the lossless reconstruction of the source component $S_{1}$ at the first receiver. Also, the inequality (14a) gives

[TABLE]

and the inequality (14c) gives

[TABLE]

for some $U_{0}$ that satisfies (15) and (16). Next, combining (21) and (23), we get

[TABLE]

The end of the proof of the claim follows by noticing that the inequalities in (22) and (24) hold with equality with the choice $U_{0}=(X_{2},X_{3})$. ∎

The rate region of Claim 1 is depicted in Figure 5. It is insightful to notice that although the second user is only interested in reproducing the component $S_{2}=X_{4}$, the optimal coding scheme that achieves this region sets the common description that is destined to be recovered by both users as one that is composed of not only $S_{2}$ but also some part $U_{0}=(X_{2},X_{3})$ of the source component $S_{1}$ (though the latter is not required by the second user). A possible intuition is that the choice $U_{0}=(X_{2},X_{3})$ is useful for user 1, who wants to reproduce $S_{1}=(X_{1},X_{2},X_{3})$, and its transmission to also the second user does not cost any rate loss since this user has available side information $Y_{2}=(X_{2},X_{3})$.

Remark 6**.**

The above observation on that it is generally beneficial that the common description to be recovered by both users contains all or part of the source component $S_{1}$ was also made in the authors’ own recent work [10] (see also [16]) in the context of a Heegard-Berger problem with degraded reconstructions sets and no successive refinement, i.e., $R_{1}=0$. In the present model, however, the intuition is less easy to grasp, because of the presence of the refinement link. For instance, the reader may wonder whether it is equally optimal that every information that is of interest for only the first user (e.g., $(X_{2},X_{3})$ in the binary example of Figure 4) be sent only over the dedicated private link of that user. Our analysis shows that while this is sometimes equally optimal, it is not in general; and, in fact, roughly speaking the optimal choice of the common description layer (and, so, of the auxiliary random variable $U_{0}$) depends on the relative strength of the pair $(R_{1},Y_{1})$ in comparison with $Y_{2}$. For example, consider again the binary example of Figure 4. For this example, the lossless transmission of the source component $S_{1}$ to the first user requires $H(S_{1}|S_{2}Y_{1})=2$ bits/sample. If the capacity of the rate-limited private link suffices to convey such information, i.e., $R_{1}\geq 2$ bits/sample, then the common-description needs not contain any part of $S_{1}$ and the choice $U_{0}=\emptyset$ is equally optimal. If, however, the rate-limited private link is not good enough to convey the required information, e.g., $1\leq R_{1}<2$ bits/sample, the encoder should transmit part of $S_{1}$ as part of the common description, e.g., through the choices $U_{0}=X_{2}$ or $U_{0}=X_{3}$ which can easily be shown to be equally optimal in this case. ∎

V The Heegard-Berger Problem with Scalable Coding

Consider now the model of Figure 3b. As we already mentioned, the reader may find it appropriate for the motivation to think about the side information $Y^{n}_{2}$ as being of lower quality than $Y^{n}_{1}$, in which case the refinement link that is given to the second user is in accordance with the motivation that the receiver with the better-quality side information (Receiver $1$) uses only the common description layer to decode while the receiver with the low-quality side information (Receiver $2$) needs the two descriptions to decode. The results that will follow, however, hold for general side information sequences.

V-A Rate-Distortion Region

The following theorem states the rate-distortion region of the Heegard-Berger model with scalable coding of Figure 3b.

Theorem 3**.**

The rate-distortion region of the Heegard-Berger model with scalable coding of Figure 3b is given by the set of all rate-distortion triples $(R_{0},R_{2},D_{1})$ that satisfy

[TABLE]

for some product pmf $P_{U_{0}U_{1}S_{1}S_{2}Y_{1}Y_{2}}$, such that:

1) the following Markov chain is valid:

[TABLE]

2) and there exists a function $\phi:\mathcal{Y}_{1}\times\mathcal{U}_{0}\times\mathcal{U}_{1}\times\mathcal{S}_{2}\rightarrow\hat{\mathcal{S}}_{1}$ such that:

[TABLE]

Proof: The proof of Theorem 3 follows from that of Theorem 1 by seeting $R_{1}=0$ therein.

Remark 7**.**

In the case in which the source component $S_{1}$ as well is recovered losslessly at Receiver $1$, the rate distortion region of Theorem 2 reduces to the set of rate pairs $(R_{0},R_{1})$ that satisfy

[TABLE]

for some product pmf $P_{U_{0}|S_{1}S_{2}}$. As we already mentioned, the result of Theorem 3, and so also the rate region described by  (28) in the lossless case, hold for general, arbitrarily correlated, side information pairs $(Y_{1},Y_{2})$. The rate region given by (28) generalizes that of [3, Theorem 4, item (1)], that is developed by Tian and Diggavi for the case in which $(S_{1},S_{2})-\!\!\!\!\minuso\!\!\!\!-Y_{1}-\!\!\!\!\minuso\!\!\!\!-Y_{2}$ forms a Markov chain, to the case of arbitrarily correlated side information pairs $(Y_{1},Y_{2})$. ∎

Remark 8**.**

In the specific case in which Receiver $2$ has a better-quality side information in the sense that $(S_{1},S_{2})-\!\!\!\!\minuso\!\!\!\!-Y_{2}-\!\!\!\!\minuso\!\!\!\!-Y_{1}$ forms a Markov chain, the rate distortion region of Theorem 3 reduces to one that is described by a single rate-constraint, namely

[TABLE]

for some conditional $P_{U|S_{1}S_{2}}$ that satisfies $\mathbb{E}[d_{1}(S_{1},\hat{S}_{1}]\leq D_{1}$. This is in accordance with the observation that, in this case, the transmission to Receiver $1$ becomes the bottleneck, as Receiver $2$ can recover the source component $S_{2}$ losslessly as long as so does Receiver $1$.

Remark 9**.**

In [15, Theorem 1], Timo et al. present an achievable rate-region for the multistage successive-refinement problem with side information. Timo et al. consider distortion measures of the form $\delta_{l}\>:\>\mathcal{X}{\times}\hat{\mathcal{X}}_{l}\rightarrow\mathbb{R}_{+}$, where $\mathcal{X}$ is the source alphabet and $\hat{\mathcal{X}}_{l}$ is the reconstruction at decoder $l$, $l\in\{1,\ldots,t\}$; and for this reason this result is not applicable as is to the setting of Figure 3b, in the case of two decoders. However, the result of [15, Theorem 1] can be extended to accomodate a distortion measure at the first decoder that is vector-valued; and the direct part of Theorem 3 can then be obtained by applying this extension. Specifically, in the case of two decoders, i.e., $t=2$, and with $X=(S_{1},S_{2})$, and two distortion measures $\delta_{1}:\mathcal{S}_{1}\times\mathcal{S}_{2}\times\hat{\mathcal{S}}_{1,1}\times\hat{\mathcal{S}}_{1,2}\rightarrow\{0,1\}\times\mathbb{R}_{+}$ and $\delta_{2}:\mathcal{S}_{1}\times\mathcal{S}_{2}\times\hat{\mathcal{S}}_{1,2}\times\hat{\mathcal{S}}_{2,2}\rightarrow\{0,1\}$ chosen such that

[TABLE]

and

[TABLE]

where $d_{H}(\cdot,\cdot)$ is the Hamming distance, letting $d_{1}=(0,D_{1})$ and $d_{2}=0$, a straightforward extension of  [15, Theorem 1] to this setting yields a rate-region that is described by the following rate constraints (using the notation of  [15, Theorem 1])

[TABLE]

where $\mathcal{T}_{0}=\{1,2\}$, $\mathcal{T}_{1}=\{1\}$, $T_{2}=\{2\}$, and for $j=0,1,2$ and $l\in{1,2}$ such that $\mathcal{T}_{j}\cap\{1,\ldots,l\}\neq\emptyset$, the function $\Phi(\mathcal{T}_{j},l)$, $j=0,1,2$, is defined as

[TABLE]

where $\mathcal{A}=\{U_{12},U_{1},U_{2}\}$ and the sets $\mathcal{A}_{\mathcal{T}_{j}}^{-}$, $\mathcal{A}_{\mathcal{T}_{j}}^{\supset}$, $\mathcal{A}_{\mathcal{T}_{j}}^{+}$, $\mathcal{A}_{\mathcal{T}_{j}}^{\dagger}$, $\mathcal{A}_{\mathcal{T}_{j},1}^{\ddagger}$, $\mathcal{A}_{\mathcal{T}_{j},2}^{\ddagger}$, evaluated in this case, are given in Table I. It is easy to see that the region described by (32) can be written more explicitly in this case as

[TABLE]

Also, setting $U_{12}=(U_{0},S_{2})$ and $U_{2}=S_{2}$ in (34) one recovers the rate-region of Theorem 3. ( Such a connection can also be stated for the result of Theorem 2 ).

V-B Binary Example

Let $X_{1}$, $X_{2}$ and $X_{3}$ be three independent $\text{Ber}(1/2)$ random variables. Let now the sources be such that $S_{1}\triangleq(X_{1},X_{3})$ and $S_{2}\triangleq X_{2}$. Now, consider the Heegard-Berger model with scalable coding shown in Figure 4. The first user gets only the common rate-limited link; and it observes the side information $Y_{1}=(X_{1},X_{2})$ and wants to reproduce the pair $(S_{1},S_{2})$ losslessly. The second user gets both the common and refinement rate-limited links; and it observes the side information $Y_{2}=X_{3}$ and wants to reproduce only the component $S_{2}$, losslessly.

For this example as well, it is easy to see that the side information at the decoders do not exhibit any degradedness ordering, in the sense that none of the Markov chain conditions of Remark 8 and Remark 7 hold. The following claim provides the rate-region of this binary example.

Claim 2**.**

The rate region of the binary Heegard-Berger example with scalable coding of Figure 6 is given by the set of all rate pairs $(R_{0},R_{1})$ that satisfy $R_{2}\geq 0$ and $R_{0}\geq 1$.

Proof: The proof of Claim 2 follows easily by specializing, and computing, the result of Theorem 3 for the example at hand, as follows. First note that for $D_{1}=0$ and the variables $(S_{1},S_{2},Y_{1},Y_{2})$ chosen as in this example, the constraint (25a) on the rate of the common link becomes

[TABLE]

where the first equality follows since $H(S_{1}|S_{2}Y_{1}U_{0}U_{1})=0$ for all random variables $U_{0}$ and $U_{1}$ that satisfy (26) and (27), by the lossless reconstruction of the source component $S_{1}$ at the first receiver. Also, the constraint (25b) on the sum of the rates on the two links becomes

[TABLE]

where the last inequality follows since $H(X_{1}|X_{2}U_{0})\leq H(X_{1})=1$ for all choices of $U_{0}$ that satisfy (26) and (27).

The end of the proof of the claim follows by noticing that the last two inequalities on the RHS of (36) hold with equality with the choice $U_{0}=X_{3}$. ∎

The optimal rate region of Claim 2 is depicted in Figure 7, as the region delimited by the lines $R_{0}=1$ and $R_{2}=0$. Note that for this example, the source component $X_{2}$, which is the only source component that is required by Receiver $2$, needs to be transmitted entirely on the common link so as to be recovered losslessly also by Receiver $1$. For this reason, the refinement link is not-constrained and appears to be useless for this example. The reader may notice here the sharp difference with binary Heegard-Berger example with successive refinement of Figure 4 for which the refinement link may sometimes be instrumental to reducing the required rate on the common link. Also, it is insightful to notice that for this example, because of the side information configuration, the choice $U_{0}=\emptyset$ in Theorem 3 is strictly suboptimal and results in the smaller region that is described by

[TABLE]

VI Proof of Theorem 1

VI-A Proof of Converse Part

Assume that a rate triple $(R_{0},R_{1},R_{2})$ is $D_{1}$-achievable. Let then $W_{j}=f_{j}(S_{1}^{n},S_{2}^{n})$, where $j\in\{0,1,2\}$, be the encoded indices and let $\hat{S}_{1}^{n}=g_{1}(W_{0},W_{1},Y_{1}^{n})$ be the reconstruction sequence at the first decoder such that $\mathds{E}d^{(n)}_{1}(S_{1}^{n},\hat{S}_{1}^{n})\leq D_{1}$.

Using Fano’s inequality, the lossless reconstruction of the source $S_{2}^{n}$ at both decoders implies that there exists a sequence $\epsilon_{n}\underset{n\rightarrow\infty}{\rightarrow}0$ such that:

[TABLE]

We start by showing the following sum-rate constraint,

[TABLE]

We have that

[TABLE]

where $(a)$ stems from (38).

Let us define then:

[TABLE]

In the following, we aim at lower-bounding each of the two quantities $A$ and $B$.

Let us start by writing that

[TABLE]

where $(a)$ follows using the following Csiszár-Körner sum-identity

[TABLE]

$(b)$ follows using the Csiszár-Körner sum-identity given by

[TABLE]

while $(c)$ is the consequence of the following sequence of Markov chains

[TABLE]

where (47a) results from that the source sequences $(S_{1}^{n},S_{2}^{n},Y_{1}^{n},Y_{2}^{n})$ are memoryless, while $(a)$ is a consequence of that $W_{0}$ is a function of the pair of sequences $(S_{1}^{n},S_{2}^{n})$.

To lower-bound the term B, note the following

[TABLE]

where $(a)$ is a consequence of the following sequence of Markov chains:

[TABLE]

where (49a) results from that the source sequences $(S_{1}^{n},S_{2}^{n},Y_{1}^{n},Y_{2}^{n})$ are memoryless, while $(a)$ is a consequence of that $W_{0}$ and $W_{1}$ are each function of the pair of sequences $(S_{1}^{n},S_{2}^{n})$.

Finally, letting $U_{1,i}\triangleq(W_{1},Y_{1}^{i-1})$ so that the choice of $(U_{0,i},U_{1,i})$ satisfy the condition: $\hat{S}_{1,i}=g_{i}(Y_{1,i},U_{0,i},U_{1,i},S_{2,i})$, we write the resulting sum-rate constraint as

[TABLE]

Let us now prove that the following bound holds

[TABLE]

We have

[TABLE]

where $(a)$ is a consequence of Fano’s inequality in (38) and $(b)$ results from the lower bound on B in (48e).

As for the third rate constraint

[TABLE]

we write

[TABLE]

where $(a)$ is a consequence of Fano’s inequality in (39) and $(b)$ stems for the following sequence of Markov Chain

[TABLE]

where (55a) results from that the source sequences $(S_{1}^{n},S_{2}^{n},Y_{1}^{n},Y_{2}^{n})$ are memoryless, while $(a)$ is a consequence of that $W_{0}$ and $W_{1}$ are each function of the pair of sequences $(S_{1}^{n},S_{2}^{n})$.

Let $Q$ be an integer-valued random variable, ranging from $1$ to $n$, uniformly distributed over $[1:n]$ and independent of all other variables $(S_{1},S_{2},U_{0},U_{1},Y_{1},Y_{2})$. We have

[TABLE]

where $(a)$ is a consequence of that all sources $(S_{1}^{n},S_{2}^{n},Y_{1}^{n},Y_{2}^{n})$ are memoryless.

Let us now define $U_{1}\triangleq(Q,U_{1,Q})$ and $U_{0}\triangleq(Q,U_{0,Q})$, we obtain

[TABLE]

The two other rate constraints can be written in a similar fashion,

[TABLE]

and this completes the proof of converse. ∎

VI-B Proof of Direct Part

For the proof of achievability of Theorem 1, for convenience 111The readers who are well acquainted with coding for Heergard-Berger type and Gray-Wyner models, may find the region more appropriate (for intuitions) in its form of Proposition 1. we first show that the rate-distortion region of the proposition that will follow is achievable. The achievability of the rate-distortion region of Theorem 1 follows by choosing the random variable $V_{0}$ of the proposition as $V_{0}=(U_{0},S_{2})$.

Proposition 1**.**

An inner bound on the rate-distortion region of the Gray-Wyner model with side information and degraded reconstruction sets of Figure 2 is given by the set of all rate-distortion quadruples $(R,R_{1},R_{2},D_{1})$ that satisfy

[TABLE]

for some choice of the random variables $(V_{0},U_{1})$ such that $(V_{0},U_{1})-\!\!\!\!\minuso\!\!\!\!-(S_{1},S_{2})-\!\!\!\!\minuso\!\!\!\!-(Y_{1},Y_{2})$ and there exist functions $g_{1}$, $g_{2,1}$, and $g_{2,2}$ such that:

[TABLE]

and

[TABLE]

Proof of Proposition 1: We now describe a coding scheme that achieves the rate-distortion region of Proposition 1. The scheme is very similar to one that is developed by Shayevitz and Wigger [4, Theorem 2] for a Gray-Wyner model with side information. In particular, similar to  [4, Theorem 2] it uses a double-binning technique for the common codebook, one that is relevant for Receiver $1$ and one that is relevant for Receiver $2$. Note, however, that, formally, the result of Proposition 1 cannot be obtained by readily applying  [4, Theorem 2] as is; and one needs to extend the result of  [4, Theorem 2] in a manner that accounts for that the source component $S^{n}_{2}$ is to be recovered losslessly by both decoders. This can be obtained by extending the distortion measure of  [4, Theorem 2] to one that is vector-valued, i.e., $d\left((s_{1},s_{2}),(\hat{s}_{1},\hat{s}_{2})\right)=\left(d_{1}(s_{1},\hat{s}_{1}),d_{H}(s_{2},\hat{s}_{2})\right)$, where $d_{H}(\cdot,\cdot)$ denotes the Hamming distance. For reasons of completeness, we provide here an outline proof of Proposition 1.

Our scheme has the following parameters: a conditional joint measure $P_{V_{0}U_{1}|S_{1}S_{2}}$ that satisfies (60) and (61), and non-negative communication rates $T_{0}$, $T_{1}$, $T_{0,0}$, $T_{0,p}$, $T_{1,0}$, $T_{1,1}$, $\tilde{R}_{0,0}$, $\tilde{R}_{0,1}$, $\tilde{R}_{0,2}$, $\tilde{R}_{1,0}$ and $\tilde{R}_{1,1}$ such that

[TABLE]

Codebook Generation

Randomly and independently generate $2^{nT_{0}}$ length-$n$ codewords $v_{0}^{n}(k_{0})$ indexed with the pair of indices $k_{0}=(k_{0,0},k_{0,p})$, where $k_{0,0}\in[1:2^{nT_{0,0}}]$ and $k_{0,p}\in[1:2^{nT_{0,p}}]$. Each codeword $v_{0}^{n}(k_{0})$ has i.i.d entries drawn according to $\displaystyle\prod_{i=1}^{n}P_{V_{0}}(v_{0,i}(k_{0}))$. The codewords $\{v_{0}^{n}(k_{0})\}$ are partitioned into superbins whose indices will be relevant for both receivers; and each superbin is partioned int two different ways, each into subbins whose indices will be relevant for a distinct receiver (i.e., double-binning). This is obtained by partitioning the indices $\{(k_{0,0},k_{0,p})\}$ as follows. We partition the $2^{nT_{0,0}}$ indices $\{k_{0,0}\}$ into $2^{n\tilde{R}_{0,0}}$ bins by randomly and independently assigning each index $k_{0,0}$ to an index $\tilde{w}_{0,0}(k_{0,0})$ according to a uniform pmf over $[1:2^{n\tilde{R}_{0,0}}]$. We refer to each subset of indices $\{k_{0,0}\}$ with the same index $\tilde{w}_{0,0}$ as a bin $\mathcal{B}_{00}(\tilde{w}_{0,0})$, $\tilde{w}_{0,0}\in[1:2^{n\tilde{R}_{0,0}}]$. Also, we make two distinct partitions of the $2^{nT_{0,p}}$ indices $\{k_{0,p}\}$, each relevant for a distinct receiver. In the first partition, which is relevant for Receiver $1$, the indices $\{k_{0,p}\}$ are assigned randomly and independently each to an index $\tilde{w}_{0,1}(k_{0,p})$ according to a uniform pmf over $[1:2^{n\tilde{R}_{0,1}}]$. We refer to each subset of indices $\{k_{0,p}\}$ with the same index $\tilde{w}_{0,1}$ as a bin $\mathcal{B}_{01}(\tilde{w}_{0,1})$, $\tilde{w}_{0,1}\in[1:2^{n\tilde{R}_{0,1}}]$. Similarly, in the second partition, which is relevant for Receiver $2$, the indices $\{k_{0,p}\}$ are assigned randomly and independently each to an index $\tilde{w}_{0,2}(k_{0,p})$ according to a uniform pmf over $[1:2^{n\tilde{R}_{0,2}}]$; and refer to each subset of indices $\{k_{0,p}\}$ with the same index $\tilde{w}_{0,2}$ as a bin $\mathcal{B}_{02}(\tilde{w}_{0,2})$, $\tilde{w}_{0,2}\in[1:2^{n\tilde{R}_{0,2}}]$.

• 2)

For each $k_{0}\in[1:2^{nT_{0}}]$, randomly and independently generate $2^{nT_{1}}$ length-$n$ codewords $u_{1}^{n}(k_{1},k_{0})$ indexed with the pair of indices $k_{1}=(k_{1,0},k_{1,1})$, where $k_{1,0}\in[1:2^{nT_{1,0}}]$ and $k_{1,1}\in[1:2^{nT_{1,1}}]$. Each codeword $u_{1}^{n}(k_{1},k_{0})$ is with i.i.d elements drawn according to $\displaystyle\prod_{i=1}^{n}P_{U_{1}|V_{0}}(u_{1,i}(k_{1},k_{0})|v_{0,i}(k_{0}))$. We partition the $2^{nT_{1,0}}$ indices $\{k_{1,0}\}$ into $2^{n\tilde{R}_{1,0}}$ bins by randomly and independently assigning each index $k_{1,0}$ to an index $\tilde{w}_{1,0}(k_{1,0})$ according to a uniform pmf over $[1:2^{n\tilde{R}_{1,0}}]$. We refer to each subset of indices $\{k_{1,0}\}$ with the same index $\tilde{w}_{1,0}$ as a bin $\mathcal{B}_{10}(\tilde{w}_{1,0})$, $\tilde{w}_{1,0}\in[1:2^{n\tilde{R}_{1,0}}]$. Similarly, we partition the $2^{nT_{1,1}}$ indices $\{k_{1,1}\}$ into $2^{n\tilde{R}_{1,1}}$ bins by randomly and independently assigning each index $k_{1,1}$ to an index $\tilde{w}_{1,1}(k_{1,1})$ according to a uniform pmf over $[1:2^{n\tilde{R}_{1,1}}]$; and refer to each subset of indices $\{k_{1,1}\}$ with the same index $\tilde{w}_{1,1}$ as a bin $\mathcal{B}_{11}(\tilde{w}_{1,1})$, $\tilde{w}_{1,1}\in[1:2^{n\tilde{R}_{1,1}}]$.

• 3)

Reveal all codebooks and its partitions to the encoder, the codebook of $\{v_{0}^{n}(k_{0})\}$ and its partitions to both receivers, and the codebook of $\{u_{1}^{n}(k_{1},k_{0})\}$ and its partitions to only Receiver $1$.

Encoding

Upon observing the source pair $(S^{n}_{1},S^{n}_{2})=(s_{1}^{n},s_{2}^{n})$, the encoder finds an index $k_{0}=(k_{0,0},k_{0,p})$ such that the codeword $v_{0}^{n}(k_{0})$ is jointly typical with $(s^{n}_{1},s^{n}_{2})$, i.e.,

[TABLE]

By the covering lemma [18, Chapter 3], the encoding in this step is successful as long as $n$ is large and

[TABLE]

Next, it finds an index $k_{1}=(k_{1,0},k_{1,1})$ such that the codeword $u_{1}^{n}(k_{1},k_{0})$ is jointly typical with the triple $(s_{1}^{n},s_{2}^{n},v_{0}^{n}(k_{0}))$, i.e.,

[TABLE]

Again, by the covering lemma [18, Chapter 3], the encoding in this step is successful as long as $n$ is large and

[TABLE]

Let $\tilde{w}_{0,0}$, $\tilde{w}_{0,1}$ and $\tilde{w}_{0,2}$ be the bin indices such that $k_{0,0}\in\mathcal{B}_{00}(\tilde{w}_{0,0})$, $k_{0,p}\in\mathcal{B}_{01}(\tilde{w}_{0,1})$ and $k_{0,p}\in\mathcal{B}_{02}(\tilde{w}_{0,2})$. Also, let $\tilde{w}_{1,0}$ and $\tilde{w}_{1,1}$ be the bin indices such that $k_{1,0}\in\mathcal{B}_{10}(\tilde{w}_{1,0})$ and $k_{1,1}\in\mathcal{B}_{11}(\tilde{w}_{1,1})$. The encoder then sends the product message $W_{0}=(\tilde{w}_{0,0},\tilde{w}_{1,0})$ over the error-free rate-limited common link of capacity $R_{0}$. Also, it sends the product message $W_{1}=(\tilde{w}_{0,1},\tilde{w}_{1,1})$ over the error-free rate-limited individual link to Receiver $1$ of capacity $R_{1}$, and the message $W_{2}=\tilde{w}_{0,2}$ over the error-free rate-limited individual link to Receiver $2$ of capacity $R_{2}$.

Decoding

Receiver $1$ gets the messages $(W_{0},W_{1})=(\tilde{w}_{0,0},\tilde{w}_{1,0},\tilde{w}_{0,1},\tilde{w}_{1,1})$. It seeks a codeword $v_{0}^{n}(k_{0})$ and a codeword $u^{n}_{1}(k_{1},k_{0})$, with the indices $k_{0}=(k_{0,0},k_{0,p})$ and $k_{1}=(k_{1,0},k_{1,1})$ satisfying $k_{0,0}\in\mathcal{B}_{00}(\tilde{w}_{0,0})$, $k_{0,p}\in\mathcal{B}_{01}(\tilde{w}_{0,1})$, $k_{1,0}\in\mathcal{B}_{10}(\tilde{w}_{1,0})$ and $k_{1,1}\in\mathcal{B}_{11}(\tilde{w}_{1,1})$, and such that

[TABLE]

By the multivariate packing lemma [18, Chapter 12], the error in this decoding step at Receiver $1$ vanishes exponentially as long as $n$ is large and

[TABLE]

Receiver $1$ then sets its reproduced codewords $\hat{s}^{n}_{2,1}$ and $\hat{s}^{n}_{1}$ respectively as

[TABLE]

Similary, Receiver $2$ gets the message $(W_{0},W_{2})=(\tilde{w}_{0,0},\tilde{w}_{1,0},\tilde{w}_{0,2})$. It seeks a codeword $v_{0}^{n}(k_{0})$, with $k_{0}=(k_{0,0},k_{0,p})$ satisfying $k_{0,0}\in\mathcal{B}_{00}(\tilde{w}_{0,0})$ and $k_{0,p}\in\mathcal{B}_{02}(\tilde{w}_{0,2})$, and such that

[TABLE]

Again, using the multivariate packing lemma [18, Chapter 12], the error in this decoding step at Receiver $2$ vanishes exponentially as long as $n$ is large and

[TABLE]

Receiver $2$ then ets its reconstructed codeword $\hat{s}^{n}_{2,1}$ as

[TABLE]

Summarizing, combining (64), (66), (68) and (71), the communication rates $T_{0}$, $T_{1}$, $T_{0,0}$, $T_{0,p}$, $T_{1,0}$, $T_{1,1}$, $\tilde{R}_{0,0}$, $\tilde{R}_{0,1}$, $\tilde{R}_{0,2}$, $\tilde{R}_{1,0}$ and $\tilde{R}_{1,1}$ satisfy the following inequalities

[TABLE]

Choosing $\tilde{R}_{0,0}$, $\tilde{R}_{1,1}$, $\tilde{R}_{0,2}$, $\tilde{R}_{1,0}$ and $\tilde{R}_{1,1}$ to also satisfy the rate relations

[TABLE]

and, finally, using Fourier-Motzkin elimination (FME) to successively project out the nuisance variables $T_{0,0}$, $T_{0,p}$, $T_{1,0}$, $T_{1,1}$, $T_{0}$, $T_{1}$, and then $\tilde{R}_{0,0}$, $\tilde{R}_{0,1}$, $\tilde{R}_{0,2}$, $\tilde{R}_{1,0}$ and $\tilde{R}_{1,1}$ from the set of relations formed by (62), (73) and (74), we get the region of Proposition 1.

This completes the proof of the proposition; and so that of the direct part of Theorem 1.