Universal Sampling Rate Distortion

TL;DR

This paper introduces universal sampling and compression schemes for correlated sources that do not require prior knowledge of source distributions, providing optimal rate-distortion trade-offs in various sampling scenarios.

Contribution

It presents the first single-letter characterizations of universal sampling rate distortion functions for different sampling methods, including fixed-set and random sampling.

Findings

Universal schemes perform robustly without knowing source distributions.

Successive sampling mechanisms improve reconstruction quality.

New joint source learning and compression schemes are developed.

Abstract

We examine the coordinated and universal rate-efficient sampling of a subset of correlated discrete memoryless sources followed by lossy compression of the sampled sources. The goal is to reconstruct a predesignated subset of sources within a specified level of distortion. The combined sampling mechanism and rate distortion code are universal in that they are devised to perform robustly without exact knowledge of the underlying joint probability distribution of the sources. In Bayesian as well as nonBayesian settings, single-letter characterizations are provided for the universal sampling rate distortion function for fixed-set sampling, independent random sampling and memoryless random sampling. It is illustrated how these sampling mechanisms are successively better. Our achievability proofs bring forth new schemes for joint source distribution-learning and lossy compression.

Equations

$$P_{S^n|X_{\mathcal{M}}^n\theta }=P_{S^n|X_{\mathcal{M}}^n}=\prod _{t=1}^n{P}_{S_t|X_{\mathcal{M}}^t{S}^{t-1}}.\text{vspace}\ast -0.1cm$$

$$P_{S_t|X_{\mathcal{M}}^t{S}^{t-1}}=P_{S_t|X_{\mathcal{M}t}},t=1,\dots ,n,$$

$$P_{S_t|X_{\mathcal{M}}^t{S}^{t-1}}=P_{S_t},t=1,\dots ,n,$$

$$P_{S_t|X_{\mathcal{M}}^t{S}^{t-1}}=\text{mathbbm}1(S_t=A),t=1,\dots ,n$$

$$\displaystyle\begin{split}\mathbb{E}\Big{[}d\Big{(}X_{B}^{n},\varphi\big{(}S^{n},f(S^{n},X_{S}^{n})\big{)}\Big{)}\Big{]}&\triangleq\mathbb{E}\Big{[}\dfrac{1}{n}\sum_{t=1}^{n}d\Big{(}X_{Bt},\Big{(}\varphi\big{(}S^{n},f(S^{n},X_{S}^{n})\big{)}\Big{)}_{t}\Big{)}\Big{]}\\ &=\sum\limits_{\tau\in\Theta}\mu_{\theta}(\tau)\mathbb{E}\Big{[}\dfrac{1}{n}\sum_{t=1}^{n}d\Big{(}X_{Bt},\Big{(}\varphi\big{(}S^{n},f(S^{n},X_{S}^{n})\big{)}\Big{)}_{t}\Big{)}\Big{|}\theta=\tau\Big{]}\vspace*{-0.4cm}\\ &\leq\Delta.\vspace*{-0.2cm}\end{split}\vspace*{-0.2cm}$$

$$\displaystyle\begin{split}\underset{\tau\in\Theta}{\max}\ \mathbbm{E}\Big{[}d\Big{(}X_{B}^{n},\varphi\big{(}S^{n},f(S^{n},X_{S}^{n})\big{)}\Big{)}\big{|}\theta=\tau\Big{]}\leq\Delta,\end{split}$$

$$P_{X_A|{\theta }_1={\tau }_1}=P_{X_A|\theta =\tau },\tau \in \Lambda ({\tau }_1).$$

$$R_A(\Delta )=\underset{\genfrac{}{}{0ex}{}{X_{\mathcal{M}}-\circ -X_A-\circ -Y_B}{\text{mathbbm}E[d(X_B,Y_B)]\le \Delta }}{\min }I(X_A\wedge Y_B),{\Delta }_{\min }\le \Delta \le {\Delta }_{\max },$$

$$\displaystyle\Delta_{\min}=\mathbbm{E}\big{[}\underset{y_{B}\in{\mathcal{Y}}_{B}}{\min}\mathbbm{E}[d(X_{B},y_{B})|X_{A}]\big{]},\quad\Delta_{\max}=\underset{y_{B}\in{\mathcal{Y}}_{B}}{\min}\big{[}\mathbbm{E}[d(X_{B},y_{B})|X_{A}]\big{]},$$

$$\tilde{d}(x_A,y_B)=\text{mathbbm}E[d(X_B,y_B)|X_A=x_A].$$

$$d_{{\tau }_1}(x_A,y_B)\triangleq \text{mathbbm}E[d(X_B,y_B)|X_A=x_A,{\theta }_1={\tau }_1];$$

$${\kappa }_A^{\mathcal{B}}(\delta ,{\tau }_1)\triangleq \{P_{\theta X_{\mathcal{M}}{Y}_B}:\theta ,X_{\mathcal{M}}-\circ -{\theta }_1,X_A-\circ -Y_B,\text{mathbbm}E[d_{{\tau }_1}(X_A,Y_B)|{\theta }_1={\tau }_1]\le \delta \},$$

$${\rho }_A^{\mathcal{B}}(\delta ,{\tau }_1)\triangleq \underset{{\kappa }_A^{\mathcal{B}}(\delta ,{\tau }_1)}{\min }I(X_A\wedge Y_B|{\theta }_1={\tau }_1)$$

$${\kappa }_A^{n\mathcal{B}}(\delta ,{\tau }_1)\triangleq \{P_{X_{\mathcal{M}}{Y}_B|\theta =\tau }=P_{X_{\mathcal{M}}|\theta =\tau }{P}_{Y_B|X_A,{\theta }_1={\tau }_1}:\text{mathbbm}E[d(X_B,Y_B)|\theta =\tau ]\le \delta ,\tau \in \Lambda ({\tau }_1)\}$$

$${\rho }_A^{n\mathcal{B}}(\delta ,{\tau }_1)\triangleq \underset{{\kappa }_A^{n\mathcal{B}}(\delta ,{\tau }_1)}{\min }I(X_A\wedge Y_B|{\theta }_1={\tau }_1).\text{vspace}\ast -0.3cm$$

$$R_A(\Delta )$$

$$\displaystyle\Delta_{\min}=\mathbbm{E}\Big{[}\mathbbm{E}[\underset{y_{B}\in{\mathcal{Y}}_{B}}{\min}d_{\theta_{1}}(X_{A},y_{B})|\theta_{1}]\Big{]}=\mathbbm{E}[\underset{y_{B}\in{\mathcal{Y}}_{B}}{\min}d_{\theta_{1}}(X_{A},y_{B})],$$

$$\displaystyle\Delta_{\max}=\mathbbm{E}\big{[}\underset{y_{B}\in{\mathcal{Y}}_{B}}{\min}\mathbbm{E}[d_{\theta_{1}}(X_{A},y_{B})|\theta_{1}]\big{]}.$$

$$R_A(\Delta )$$

$${\Delta }_{\min }=\underset{{\tau }_1\in {\Theta }_1}{\max }\underset{P_{Y_B|X_A,{\theta }_1={\tau }_1}=P_{Y_B|X_{\mathcal{M}},\theta =\tau }}{\min }\underset{\tau \in \Lambda ({\tau }_1)}{\max }\text{mathbbm}E[d(X_B,Y_B)|\theta =\tau ]$$

$${\Delta }_{\max }=\underset{{\tau }_1\in {\Theta }_1}{\max }\underset{y_B\in {\mathcal{Y}}_B}{\min }\underset{\tau \in \Lambda ({\tau }_1)}{\max }\text{mathbbm}E[d(X_B,y_B)|\theta =\tau ].$$

$$\underset{{\tau }_1\in {\Theta }_1}{\max }{\rho }_A^{\mathcal{B}}({\Delta }_{{\tau }_1},{\tau }_1)$$

$${\rho }_A^{\mathcal{B}}({\Delta }_{{\tau }_1}^{\ast },{\tau }_1)\equiv \underset{{\tilde{\tau }}_1\in {\Theta }_1}{\max }{\rho }_A^{\mathcal{B}}({\Delta }_{{\tilde{\tau }}_1}^{\ast },{\tilde{\tau }}_1)$$

$${\Delta }_{{\tau }_1}^{\ast }=\text{mathbbm}E[\underset{y_B\in {\mathcal{Y}}_B}{\min }{d}_{{\tau }_1}(X_A,y_B)|{\theta }_1={\tau }_1].$$

$$d(x_B,y_B)=\text{mathbbm}1(x_B\ne y_B)=1-\prod _{i\in B}\text{mathbbm}1(x_i=y_i),x_B,y_B\in {\mathcal{X}}_B={\mathcal{Y}}_B$$

$${\rho }_A^{\mathcal{B}}({\Delta }_{{\tau }_1},{\tau }_1)=\underset{\text{mathbbm}E[{\alpha }_{{\tau }_1}(X_A)\text{mathbbm}1(X_A\ne Y_A)|{\theta }_1={\tau }_1]\le {\Delta }_{{\tau }_1}-(1-\text{mathbbm}E[{\alpha }_{{\tau }_1}(X_A)|{\theta }_1={\tau }_1])}{\min }I(X_A\wedge Y_A|{\theta }_1={\tau }_1)$$

$${\alpha }_{{\tau }_1}(x_A)=\underset{\tilde{x}\in {\mathcal{X}}_B}{\max }{P}_{X_B|X_A{\theta }_1}(\tilde{x}|x_A,{\tau }_1)$$

$$\displaystyle\Delta_{\min}=1-\mathbbm{E}[\alpha_{\theta_{1}}(X_{A})]\quad\text{ and }\quad\Delta_{\max}=1-\mathbbm{E}\big{[}\underset{x_{B}\in{\mathcal{X}}_{B}}{\max}\ P_{X_{B}|\theta_{1}}(x_{B}|\theta_{1})\big{]}.$$

$${\tilde{d}}_{{\hat{\tau }}_1}(x_A,y_A)\triangleq {\alpha }_{{\hat{\tau }}_1}(x_A)\text{mathbbm}1(x_A\ne y_A)$$

$${\Delta }_{{\hat{\tau }}_1}-(1-\text{mathbbm}E[{\alpha }_{{\hat{\tau }}_1}(X_A)|{\theta }_1={\hat{\tau }}_1]).$$

Full text

††footnotetext: *†*V. P. Boda and P. Narayan are with the Department of Electrical and Computer Engineering and the Institute for Systems Research, University of Maryland, College Park, MD 20742, USA. E-mail: {praneeth, prakash}@umd.edu. Part of this work was presented at the International Symposium on Information Theory, Aachen, Germany, 2017. This work was supported by the U.S. National Science Foundation under Grants CCF-0917057 and CCF-1319799.

I Introduction

Consider a set ${\mathcal{M}}$ of $m$ discrete memoryless sources with joint probability mass function (pmf) known only to belong to a given family of pmfs. At time instants $t=1,\ldots,n,$ possibly different subsets $A_{t}$ of size $k\leq m$ are sampled “spatially” and compressed jointly by a (block) source code, with the objective of reconstructing a predesignated subset $B\subseteq{\mathcal{M}}$ of sources from the compressed representations within a specified level of distortion. In forming an efficient rate distortion code that yields the best compression rate for a given distortion level, what are the tradeoffs – under optimal processing – among causal sampling procedure, inferential methods for approximating the underlying joint pmf of the memoryless sources, compression rate and distortion level? “Universality” requires that the combined sampling mechanism and lossy compression code be fashioned in the face of imprecise knowledge of the underlying pmf. This paper is a progression of our work in [3] on sampling rate distortion for multiple sources with known joint pmf. Motivating applications include in-network computation [8], dynamic thermal management in multicore processor chips [29], etc.

The study of problems of combined sampling and compression has a classical and distinguished history. Recent relevant works include the lossless compression of analog sources in an information theoretic setting [27]; compressed sensing with an allowed detection error rate or quantization distortion [21]; sub-Nyquist temporal sampling followed by lossy reconstruction [11]; and rate distortion function for multiple sources with time-shared sampling [17]. See also [9, 23]. Closer to our approach that entails spatial sampling, the rate distortion function has been characterized when multiple Gaussian signals from a random field are sampled and quantized (centralized or distributed) in [19]. In a setting of distributed acoustic sensing and reconstruction, centralized as well as distributed coding schemes and sampling lattices are studied in [12]. In [10], a Gaussian random field on the interval $[0,1]$ and i.i.d. in time, is reconstructed from compressed versions of $k$ sampled sequences under a mean-squared error distortion criterion. All the sampling problems above assume a knowledge of the underlying pmf.

In the realm of rate distortion theory where a complete knowledge of the signal statistics is unknown, there is a rich literature that considers various formulations of universal coding; only a sampling is listed here. Directions include classical Bayesian and nonBayesian methods [30, 18, 20, 22]; “individual sequences” studies [31, 25, 26]; redundancy in quantization rate or distortion [15, 13, 14]; and lossy compression of noisy or remote signals [16, 24, 6]. These works propose a variety of distortion measures to investigate universal reconstruction performance.

Our work differs materially from the approaches above. Sampling is spatial rather than temporal, unlike in most of the settings above. Furthermore, we introduce new forms of randomized sampling that can depend on the observed source realizations, and which yield a clear gain in performance. We restrict ourselves to universality that involves a lack of specific knowledge of source pmf within a finite family of pmfs. Accordingly, in Bayesian and nonBayesian settings, we consider average and peak distortion criteria, respectively, with an emphasis on the former. Extensions to an infinite family of pmfs are currently under study.

Our technical contributions are as follows. In Bayesian and nonBayesian settings, we consider a new formulation involving an universal sampling rate distortion function (USRDf), with the objective of capturing the interplay and characterizing inherent tradeoffs among sampling mechanism, approximation of underlying (unknown) pmf, lossy compression rate and distortion level. Our results build on the concept of sampling rate distortion function [3], which in turn uses as an ingredient the rate distortion function for a “remote” source-receiver model with known pmf [7, 1, 2, 28]. We begin with fixed-set sampling where the encoder observes the same set of $k$ sampled sources at every time instant. Recognizing that only the $k$-marginal pmf of the sources – pertaining to the sampling set – can be learned by the encoder, the corresponding USRDf is characterized. In general, allowing randomization in sampling affords two distinct advantages over fixed-set sampling: better approximation of the underlying joint pmf and improved compression performance enabled by sampling different subsets of sources in apposite proportions. An independent random sampler chooses different $k$-subsets of the sources independently of source realizations and independently in time, and can learn all $k$-marginals of the joint pmf. This reduction in pmf uncertainty (vis-à-vis fixed-set sampling) aids in improving USRDf. Interestingly, our achievability proof shows how this USRDf can be attained without informing the decoder explicitly of the sampling sequence. Lastly, we consider a more powerful sampler, namely the memoryless random sampler, whose choice of sampling sets can depend on instantaneous source realizations. Surprisingly, this latitude allows the encoder to learn the entire joint pmf, and that, too, only from the sampling sequence without recourse to the sampled source realizations. Furthermore, we show how USRDf can be attained by means of a sampling sequence that depends deterministically on source realizations, thereby reducing code complexity. Thus, all our achievability proofs bring out new ideas for joint source pmf-learning and lossy compression.

Our model is described in Section II. The main results, illustrated by examples, are stated in Section III. In Section IV, we present the achievability proofs in the increasing order of sampler complexity, with an emphasis on the Bayesian setting; a unified converse proof is presented thereafter.

II Preliminaries

Denote ${\mathcal{M}}=\{1,\ldots,m\}$, and let $X_{{\mathcal{M}}}=(X_{1},\ldots,X_{m})$ be a {\mathcal{X}}_{{\mathcal{M}}}=\mathop{\mbox{\large\times}}\limits_{i=1}^{m}{\mathcal{X}}_{i}-valued rv where each ${\mathcal{X}}_{i}$ is a finite alphabet. For a (nonempty) set $A\subseteq{\mathcal{M}}$, we denote by $X_{A}$ the rv $(X_{i},i\in A)$ with values in \mathop{\mbox{\large\times}}\limits_{i\in A}{\mathcal{X}}_{i}, and denote $n$ repetitions of $X_{A}$ by $X_{A}^{n}=(X_{i}^{n},i\in A)$ with values in {\mathcal{X}}_{A}^{n}=\mathop{\mbox{\large\times}}\limits_{i\in A}{\mathcal{X}}_{i}^{n}, where $X_{i}^{n}=(X_{i1},\ldots,X_{in})$ takes values in the $n$-fold product space ${\cal X}_{i}^{n}={\mathcal{X}}_{i}\times\cdots\times{\mathcal{X}}_{i}$. For $1\leq k\leq m$, let ${\cal A}_{k}=\{A:A\subseteq{\mathcal{M}},\ |A|=k\}$ be the set of all $k$-sized subsets of ${\mathcal{M}}$ and let $A^{c}={\mathcal{M}}\!\setminus\!A$. Let {\mathcal{Y}}_{{\mathcal{M}}}=\mathop{\mbox{\large\times}}\limits_{i=1}^{m}{\mathcal{Y}}_{i}, where ${\mathcal{Y}}_{i}$ is a finite reproduction alphabet for $X_{i}$. All logarithms and exponentiations are with respect to the base 2.

Let $\Theta$ be a finite set (of parameters) and $\theta$ a $\Theta$-valued rv with pmf $\mu_{\theta}$ of assumed full support. We consider a discrete memoryless multiple source (DMMS) $\{X_{{\mathcal{M}}t}\}_{t=1}^{\infty}$ consisting of i.i.d. repetitions of the rv $X_{{\mathcal{M}}}$ with pmf known only to the extent of belonging to a finite family of pmfs ${\mathcal{P}}=\{P_{X_{{\mathcal{M}}}|\theta=\tau},\ \tau\in\Theta\}$ of assumed full support. Two settings are studied: in a Bayesian formulation, the pmf $\mu_{\theta}$ is taken to be known while in a nonBayesian formulation $\theta$ is an unknown constant in $\Theta$.

Definition 1**.**

In the Bayesian setting, a $k$-random sampler ($k$-RS), $1\leq k\leq m$, collects causally at each $t=1,\ldots,n$, random samples $X_{S_{t}}\triangleq X_{S_{t}t}$ from $X_{{\mathcal{M}}t}$, where $S_{t}$ is a rv with values in ${\mathcal{A}}_{k}$ with (conditional) pmf $P_{S_{t}|X_{{\mathcal{M}}}^{t}S^{t-1}}$, with $X_{{\mathcal{M}}}^{t}=(X_{{\mathcal{M}}1},\ldots,X_{{\mathcal{M}}t})$ and $S^{t-1}=(S_{1},\ldots,S_{t-1})$. Such a $k$-RS is specified by a (conditional) pmf $P_{S^{n}|X_{{\mathcal{M}}}^{n}\theta}$ with the requirement

[TABLE]

In the nonBayesian setting, the first equality above is redundant. In both settings, a $k$-RS is unaware of the underlying pmf of the DMMS.

The output of a $k$-RS is $(S^{n},X_{S}^{n})$ where $X_{S}^{n}=(X_{S_{1}},\ldots,X_{S_{n}})$. Successively restrictive choices of a $k$-RS in (1) corresponding to

[TABLE]

[TABLE]

and, for a given $A\subseteq{\mathcal{M}},$

[TABLE]

will be termed the $k$-memoryless random sampler, $k$-independent random sampler and the $k$-fixed-set sampler abbreviated as $k$-MRS, $k$-IRS and $k$-FS, respectively.

Our objective is to reconstruct a subset of DMMS components with indices in an arbitrary but fixed recovery set $B\subseteq{\mathcal{M}}$, namely $X_{B}^{n},$ from a compressed representation of the $k$-RS output $(S^{n},X_{S}^{n}),$ under a suitable distortion criterion.

Definition 2**.**

An $n$-length block code with $k$-RS for a DMMS $\{X_{{\mathcal{M}}t}\}_{t=1}^{\infty}$ with alphabet ${\mathcal{X}}_{{\mathcal{M}}}$ and reproduction alphabet ${\mathcal{Y}}_{B}$ is a triple $(P_{S^{n}|X_{{\mathcal{M}}}^{n}},f_{n},\varphi_{n})$ where $P_{S^{n}|X_{{\mathcal{M}}}^{n}}$ is a $k$-RS as in (1), and $(f_{n},\varphi_{n})$ are a pair of mappings where the encoder $f_{n}$ maps the $k$-RS output $(S^{n},X_{S}^{n})$ into some finite set ${\mathcal{J}}=\{1,\ldots,J\}$ and the decoder $\varphi_{n}$, with access to $S^{n}$ and the encoder output, maps ${\mathcal{A}}_{k}^{n}\times{\mathcal{J}}$ into ${\mathcal{Y}}_{B}^{n}$. We shall use the compact notation $(P_{S|X_{{\mathcal{M}}}},f,\varphi),$ suppressing $n$. The rate of the code with $k$-RS $(P_{S|X_{{\mathcal{M}}}},f,\varphi)$ is $\tfrac{1}{n}\log||f||=\tfrac{1}{n}\log J$. (An encoder that operates by forming first an explicit estimate of $\theta$ from $(S^{n},X_{S}^{n})$ is subsumed by this definition.)

Remark: We note that the decoder $\varphi$ above is taken to be informed of the sequence of sampling sets $S^{n}$. This assumption is meaningful for a $k$-IRS and $k$-MRS. For a $k$-IRS, it will be shown to be not needed.

For a given (single-letter) finite-valued distortion measure $d:{\mathcal{X}}_{B}\times{\mathcal{Y}}_{B}\rightarrow\mathbb{R}^{+}\cup\{0\}$, an $n$-length block code with $k$-RS $(P_{S|X_{{\mathcal{M}}}},f,\varphi)$ will be required to satisfy one of the following distortion criteria $(d,\Delta)$ depending on the setting.

(i) Bayesian: The expected distortion criterion is

[TABLE]

(ii) NonBayesian: The peak distortion criterion is

[TABLE]

where the “conditional” expectation denotes, in fact, $\mathbbm{E}_{P_{X_{{\mathcal{M}}}^{n}S^{n}|\theta=\tau}}=\mathbbm{E}_{P_{X_{{\mathcal{M}}}^{n}|\theta=\tau}P_{S^{n}|X_{{\mathcal{M}}}^{n}}}.$

Definition 3**.**

A number $R\geq 0$ is an achievable universal $k$-RS coding rate at distortion level $\Delta$ if for every $\epsilon>0$ and sufficiently large $n$, there exist $n$-length block codes with $k$-RS of rate less than $R+\epsilon$ and satisfying the distortion criterion $(d,\Delta+\epsilon)$ in (LABEL:eq:expected-distortion) or (6) above; and $(R,\Delta)$ will be termed an achievable universal $k$-RS rate distortion pair under the expected or peak distortion criterion. The infimum of such achievable rates is denoted by $R_{A}(\Delta)$, $R_{\imath}(\Delta)$ and $R_{m}(\Delta)$ for a $k$-FS, $k$-IRS and $k$-MRS, respectively. We shall refer to $R_{A}(\Delta),\ R_{\imath}(\Delta)$ as well as $R_{m}(\Delta)$ as the universal sampling rate distortion function (USRDf), suppressing the dependence on $k$.

Remark: Clearly, the USRDf under (LABEL:eq:expected-distortion) will be no larger than that under (6).

III Main Results

We make the following main contributions. First, a (single-letter) characterization is provided of the USRDf for fixed-set sampling, i.e., $k$-FS, in the Bayesian and nonBayesian settings. Second, building on this, a characterization of the USRDf is obtained for a $k$-IRS in these settings, and it is shown that randomized sampling can outperform strictly the “best” fixed-set sampler. Indeed, this USRDf can be attained even upon dispensing with the a priori assumption that the decoder is informed of the sequence of sampling sets. Finally, the USRDf for a $k$-MRS is characterized and shown to be achievable by a sampler that is determined by the instantaneous realizations of the DMMS at each time instant. We note that the USRDfs for a $k$-FS and $k$-IRS can be deduced from that of a $k$-MRS. Nevertheless, for the sake of expository convenience, we develop the three sampling models in succession; this will also facilitate the presentation of the achievability proofs.

Throughout this paper, a salient theme that recurs is this: An encoder without prior knowledge of $\theta$ and with access to only $k$ instantaneously sampled components of the DMMS $\{X_{{\mathcal{M}}t}\}_{t=1}^{\infty}$ can form only a limited estimate of $\theta.$ The quality of said estimate improves steadily from $k$-FS to $k$-IRS to $k$-MRS.

Consider first fixed-set sampling with $A\subseteq{\mathcal{M}}$ in (4). An encoder $f$ with access to $X_{A}^{n}$ cannot distinguish among pmfs in ${\mathcal{P}}$ (indexed by $\tau$) that have the same $P_{X_{A}|\theta=\tau}.$ Accordingly, let $\Theta_{1}$ be a partition of $\Theta$ comprising “ambiguity” atoms, with each such atom consisting of $\tau$s with identical marginal pmfs $P_{X_{A}|\theta=\tau}$. Indexing the elements of $\Theta_{1}$ by $\tau_{1}$, let $\theta_{1}$ be a $\Theta_{1}$-valued rv with pmf $\mu_{\theta_{1}}$ induced by $\mu_{\theta}.$ For each $\tau_{1}\in\Theta_{1},$ let $\Lambda(\tau_{1})$ be the collection of $\tau$s in the atom of $\Theta_{1}$ indexed by $\tau_{1}$. In the Bayesian setting, clearly

[TABLE]

In the nonBayesian setting, in order to retain the same notation, we choose $P_{X_{A}|\theta_{1}=\tau_{1}}$ to be the right-side above.

When the pmf of the DMMS $\{X_{{\mathcal{M}}t}\}_{t=1}^{\infty}$ is known, say $P_{X_{{\mathcal{M}}}}$ – corresponding to $|\Theta|=1$ – we recall from [3] that the (U)SRDf for fixed $A\subseteq{\mathcal{M}}$ is

[TABLE]

with

[TABLE]

which can be interpreted as the (standard) rate distortion function for the DMMS $\{X_{At}\}_{t=1}^{\infty}$ using a modified distortion measure ${\tilde{d}}$ defined by

[TABLE]

This fact will serve as a stepping stone to our analysis of USRDf for a $k$-random sampler. In the Bayesian setting, we consider a modified distortion measure $d_{\tau_{1}}$, $\tau_{1}\in\Theta_{1}$, given by

[TABLE]

the set of (constrained) pmfs

[TABLE]

and the (minimized) conditional mutual information

[TABLE]

which is akin to (8) and will play a basal role. In the nonBayesian setting, the counterparts of (12) and (13) are

[TABLE]

and

[TABLE]

Remarks: (i) The minima in (13) and (15) exist as those of convex functions over convex, compact sets.

(ii) Clearly, the minimum in (15) under pmf-wise constraints (14) can be no smaller than that in (13) under pmf-averaged constraints (12).

(iii) It is seen in a standard manner that $\rho_{A}^{{\mathcal{B}}}(\delta,\tau_{1})$ in (13) and $\rho_{A}^{n{\mathcal{B}}}(\delta,\tau_{1})$ in (15) are convex and continuous in $\delta.$

Our first main result states that the USRDf at distortion level $\Delta$ for fixed-set sampling in the Bayesian setting is a minmax of quantities in (13), where the maximum is over ambiguity atoms $\tau_{1}$ in $\Theta_{1}$, while the minimum is over distortion thresholds $\delta=\Delta_{\tau_{1}},\ \tau_{1}\in\Theta_{1}$ whose mean does not exceed $\Delta$. On the other hand, in the nonBayesian setting, the USRDf at distortion level $\Delta$ is a maximum over ambiguity atoms of quantities in (15) with $\delta=\Delta,$ and hence is no smaller than its Bayesian counterpart.

Theorem 1**.**

The Bayesian USRDf for fixed $A\subseteq{\mathcal{M}}$ is

[TABLE]

for $\ \Delta_{\min}\leq\Delta\leq\Delta_{\max},$ where

[TABLE]

[TABLE]

The nonBayesian USRDf is

[TABLE]

where

[TABLE]

and

[TABLE]

Remarks: (i) In fact, the minimizing pmf $P_{Y_{B}|X_{A}\theta_{1}}$ in $\Delta_{\min}$ is a conditional point-mass.

(ii) We note that for a given distortion level $\Delta,$ the set $\{\Delta_{\tau_{1}},\ \tau_{1}\in\Theta_{1}:\ \sum\limits_{\tau_{1}\in\Theta_{1}}\mu_{\theta_{1}}(\tau_{1})\Delta_{\tau_{1}}\leq\Delta\}$ is a convex, compact set in $\mathbbm{R}^{|\Theta_{1}|}$. Next, observing that

[TABLE]

is a convex function of $\{\Delta_{\tau_{1}},\ \tau_{1}\in\Theta_{1}\}$, the minimum in (16) exists as that of a convex function over a convex, compact set.

(iii) The minimizing $\{\Delta_{\tau_{1}}^{*},\ \tau_{1}\in\Theta_{1}\}$ in (16) is characterized by the following special property: For a given $\Delta_{\min}\leq\Delta\leq\Delta_{\max},$ for each $\tau_{1}\in\Theta_{1},$ either

[TABLE]

where the right-side does not depend on $\tau_{1}$, or

[TABLE]

By a standard argument in convex optimization, if $\{\Delta_{\tau_{1}}^{*},\ \tau_{1}\in\Theta_{1}\}$ does not satisfy the property above, then a small perturbation decreases the maximum in (23) leading to a contradiction.

(iv) The $\Delta_{\min}$ and $\Delta_{\max}$ for the Bayesian and the nonBayesian settings can be different.

Example 1**.**

For the probability of error distortion measure

[TABLE]

the Bayesian USRDf for fixed-set sampling with $A\subseteq B$ in (16) simplifies with (13) becoming

[TABLE]

where

[TABLE]

is the maximum a posteriori (MAP) estimate of $X_{B}$ on the basis of $X_{A}=x_{A}$ under pmf $P_{X_{{\mathcal{M}}}|\theta_{1}=\tau_{1}}$.

The proof of (26), (27) is along the lines of that of ([3], Proposition 1) under the pmf $P_{X_{{\mathcal{M}}}|\theta_{1}=\tau_{1}}$ (rather than $P_{X_{{\mathcal{M}}}}$ as in [3]), and so is not repeated here. Furthermore,

[TABLE]

The form of the Bayesian USRDf in (26) suggests a simple achievability scheme comprising two steps. Using a maximum a posteriori (MAP) or maximum likelihood (ML) estimate ${\widehat{\tau}}_{1}$ of $\theta_{1}$ on the basis of $X_{A}^{n}=x_{A}^{n}$, the first step entails a lossy reconstruction of $x_{A}^{n}$ by its codeword $y_{A}^{n}$, under pmf $P_{X_{{\mathcal{M}}}|\theta_{1}={\widehat{\tau}}_{1}}$ and for a modified distortion measure

[TABLE]

with a corresponding reduced threshold

[TABLE]

This is followed by a second step of reconstructing $x_{B}^{n}$ from the output $y_{A}^{n}$ of the previous step as a MAP estimate

[TABLE]

the corresponding probability of estimation error coincides with the mentioned reduction $1-\mathbbm{E}[\alpha_{{\widehat{\tau}}_{1}}(X_{A})|\theta_{1}={\widehat{\tau}}_{1}]$ in the threshold.

In the nonBayesian setting, the USRDf in (19), (15) simplifies with

[TABLE]

for $\Delta_{\min}\leq\Delta\leq\Delta_{\max}$, where

[TABLE]

and

[TABLE]

This leads to the following achievability scheme. With ${\widehat{\tau}}_{1}$ as the ML estimate of $\theta_{1}$ formed from $X_{A}^{n}=x_{A}^{n}$, first $x_{A}^{n}$ is reconstructed as $y_{A}^{n}$ according to $P_{Y_{A}|X_{A},\theta_{1}={\widehat{\tau}}_{1}}$ resulting from the minimization in (32). This is followed by the reconstruction of $x_{B}^{n}$ from $y_{A}^{n}$ by means of the estimate

[TABLE]

under pmf $P_{Y_{B}|Y_{A}\theta_{1}}$ which, too, is obtained from the minimization in (32).

∎

Example 2**.**

Let ${\mathcal{M}}=\{1,2\}$ and ${{\mathcal{X}}}_{1}={\mathcal{X}}_{2}=\{0,1\},$ consider a DMMS with $P_{X_{1}X_{2}|\theta=\tau}$ represented by a virtual binary symmetric channel (BSC) shown in Figure 2, where $p_{\tau},q_{\tau}\leq 0.5,\ \tau\in\Theta$, where $\Theta$ is a given finite set. For $A=\{1\},\ B=\{1,2\}$, and the probability of error distortion measure of Example 1, the Bayesian USRDf reduces to

[TABLE]

for $\Delta_{\min}\leq\Delta\leq\Delta_{\max}$, where

[TABLE]

and $q_{\tau_{1}}=P_{X_{2}|X_{1}\theta_{1}}(0|1,\tau_{1}),\ \tau_{1}\in\Theta_{1}$; and the nonBayesian USRDf is

[TABLE]

with

[TABLE]

∎

Example 3**.**

This example, albeit concocted, shows that for fixed-set sampling with $A$ and recovery set $B$, a choice of $A$ outside $B$ can be best. Let ${\mathcal{M}}=\{1,2,3\},\ B=\{1,2\}$ and ${{\mathcal{X}}}_{i}={\mathcal{Y}}_{j}=\{0,1\},\ i=1,2,3;j=1,2.$ Consider a DMMS with $P_{X_{1}X_{2}|\theta=\tau}$ as in Figure 2 and $X_{3}=X_{1}\oplus X_{2}$ where $\oplus$ denotes addition modulo 2. Here, $p_{\tau}=0.5,\ q_{\tau}\leq 0.5,\ \tau\in\Theta$, with the $q_{\tau}$s$,\ \tau\in\Theta$, being distinct. For distortion measure $d(x_{B},y_{B})\triangleq\mathbbm{1}\left((x_{1}\oplus x_{2})\neq(y_{1}\oplus y_{2})\right)$, the Bayesian USRDf for fixed-set sampling is

[TABLE]

where ${\tilde{q}}=\sum\limits_{\tau\in\Theta}\mu_{\theta}(\tau)q_{\tau}$. Since $P_{X_{1}|\theta=\tau}$ is the same for all $\tau\in\Theta,$ note that $|\Theta_{1}|=1.$ The nonBayesian USRDf is

[TABLE]

Also, $R_{\{1\}}(\Delta)=R_{\{2\}}(\Delta)$. For sampling set $A=\{3\}$, $\Theta_{1}=\Theta$ and the Bayesian USRDf is

[TABLE]

and the nonBayesian USRDf is

[TABLE]

Clearly, $R_{\{3\}}(\Delta)\leq R_{\{1\}}(\Delta)$, with the inequality being strict for suitable values of $\Delta.$ ∎

Turning to a $k$-IRS in (3), the freedom now given to the sampler to rove over all $k$-sized subsets in ${\mathcal{A}}_{k}$ engenders a partition $\Theta_{2}$ of $\Theta_{1}$ (and hence a finer partition of $\Theta$) with smaller ambiguity atoms. Let $A_{1},\ldots,A_{|{\mathcal{A}}_{k}|},$ where $|{\mathcal{A}}_{k}|={m\choose k}$, be any fixed ordering of ${\mathcal{A}}_{k}.$ Let $\Theta_{2}$ be a partition of $\Theta$ consisting of ambiguity atoms, with each atom formed by $\tau$s with identical (ordered) collections of marginal pmfs $\left(P_{X_{A_{i}}|\theta=\tau},\ i=1,\ldots,|{\mathcal{A}}_{k}|\right).$

Clearly, $\Theta_{2}$ is a refinement of $\Theta_{1}$ (for any $A_{i}$). Indexing the elements of $\Theta_{2}$ by $\tau_{2},$ let $\theta_{2}$ be a $\Theta_{2}$-valued rv with pmf $\mu_{\theta_{2}}$ derived from $\mu_{\theta}$. For each $\tau_{2}$ in $\Theta_{2},$ let $\Lambda(\tau_{2})$ be the collection of $\tau$s in the atom indexed by $\tau_{2}$. In analogy with (13) and (15), we define counterparts in the Bayesian and nonBayesian settings as

[TABLE]

where $d_{\tau_{2}}$ is defined as in (11) with $\theta_{2}=\tau_{2}$ replacing $\theta_{1}=\tau_{1},$ and

[TABLE]

[TABLE]

Theorem 2**.**

The Bayesian USRDf for a $k$-IRS is

[TABLE]

where

[TABLE]

The nonBayesian USRDf is

[TABLE]

for

[TABLE]

and

[TABLE]

Corollary**.**

The USRDfs in the Bayesian and nonBayesian settings remain unchanged upon a restriction to $n$-length block codes $(f,\varphi)$ with uninformed decoder, i.e., with $\varphi=\varphi(f(S^{n},X_{S}^{n}))$.

Remark: (i) For a $k$-IRS we restrict ourselves to the interesting case of $k<|B|,$ for otherwise it would suffice to choose $S_{t}=B,\ t=1,\ldots,n.$

(ii) Akin to a $k$-FS, the optimizing $P_{S},\ \{\Delta_{\tau_{2}}^{*},\ \tau_{2}\in\Theta_{2}\}$ in (47) has the following special property: For a given $\Delta_{\min}\leq\Delta\leq\Delta_{\max},$ for each $\tau_{2}\in\Theta_{2},$ either

[TABLE]

or

[TABLE]

(iii) In general, a $k$-IRS will outperform a $k$-FS in two ways. First, the former enables a better approximation of $\theta$ in the form of $\theta_{2}$ whereas the latter estimates $\theta_{1}=\theta_{1}(\theta_{2})$. Second, random sampling enables a “time-sharing” over various fixed-set samplers, that can outperform strictly the best fixed-set choice. Both these advantages of a $k$-IRS over fixed-set sampling are illustrated in Examples 4 and 5.

Example 4**.**

This example illustrates that a $k$-IRS can perform strictly better than the best $k$-FS. For ${\mathcal{M}}=B=\{1,2\}$, and ${\mathcal{X}}_{i}={\mathcal{Y}}_{i}=\{0,1\},\ i=1,2,$ consider a DMMS with $P_{X_{1}X_{2}|\theta=\tau}=P_{X_{1}|\theta=\tau}P_{X_{2}|\theta=\tau}$ where

[TABLE]

and $0<p_{\tau},\ q_{\tau}<0.5$. Under the distortion measure $d(x_{B},y_{B})=\mathbbm{1}(x_{1}\neq y_{1})+\mathbbm{1}(x_{2}\neq y_{2})$, for a $k$-FS, with $k=1$, the Bayesian USRDf for sampling set $A=\{1\}$ is

[TABLE]

where $q_{\tau_{1}}=\mathbbm{E}[q_{\theta}|\theta_{1}=\tau_{1}]$, and the nonBayesian USRDf is

[TABLE]

Turning to a $k$-IRS with $k=1$, clearly, $\Theta_{2}=\Theta$. For a $k$-IRS the Bayesian USRDf is

[TABLE]

and the nonBayesian USRDf is

[TABLE]

where $I$ equals

[TABLE]

An analytical comparison of the USRDfs shows the strict superiority of the $k$-IRS over the $k$-FS, as seen – for instance – by the lower values of $\Delta_{\min}$ for the former. ∎

Example 5**.**

In Example 4, assume that

[TABLE]

For a $k$-FS with $k=1$, the nonBayesian USRDf is

[TABLE]

Now, observe that for each $\tau\in\Theta$

[TABLE]

holds for $p_{\tau}\leq\delta\leq p_{\tau}+q_{\tau}$. Thus, for a $k$-IRS with $k=1$, the nonBayesian USRDf in (56) simplifies to

[TABLE]

which is strictly smaller than the USRDf for the better $k$-FS in (58). The superior performance of the $k$-IRS is enabled by its ability to estimate simultaneously both $P_{X_{1}|\theta}$ and $P_{X_{2}|\theta}$ (and thereby $P_{X_{1}X_{2}|\theta}$); a $k$-FS can estimate only one of $P_{X_{1}|\theta}$ or $P_{X_{2}|\theta}$. ∎

Lastly, for a $k$-MRS in (2), the ability of the sampler to depend instantaneously on the current realization of the DMMS enables an encoder with access to the sampler output to distinguish among all the pmfs in ${\mathcal{P}}.$ Accordingly, for a $k$-MRS, $\Theta$ itself serves as the counterpart of the partitions $\Theta_{1}$ (for a $k$-FS) and $\Theta_{2}$ for a $k$-IRS. For a rv $U$ with fixed pmf $P_{U}$ on some finite set ${\mathcal{U}},$ and for fixed $P_{S|X_{{\mathcal{M}}}U}$, we define the counterparts of (42) and (43) as

[TABLE]

and

[TABLE]

where the minimization in (61) and (62), in effect, is with respect to $P_{Y_{B}|SX_{S}U\theta}$ and the sets of (constrained) pmfs are

[TABLE]

and

[TABLE]

Here, $U$ plays the role of a “time-sharing” rv, as will be seen below.

Theorem 3**.**

For a $k$-MRS, the Bayesian USRDf is

[TABLE]

where

[TABLE]

The nonBayesian USRDf is

[TABLE]

where

[TABLE]

and

[TABLE]

It suffices to take $|{\mathcal{U}}|\leq 2|\Theta|+1.$

In (66) and (68), (69), it is readily seen that conditionally deterministic samplers (defined below) attain the minima in $\Delta_{\min}$ and $\Delta_{\max}$. In fact, such samplers will be seen to be optimal for every $\Delta_{\min}\leq\Delta\leq\Delta_{\max}$.

For a mapping $w:{\cal X}_{{\mathcal{M}}}\times{\cal U}\rightarrow{\cal A}_{k}$, a deterministic sampler is specified in terms of a conditional point-mass pmf

[TABLE]

Theorem 3 is equivalent to

Proposition 4**.**

For a $k$-MRS, the Bayesian USRDf is

[TABLE]

with $\Delta_{\min}$ and $\Delta_{\max}$ as in (66), and the nonBayesian USRDf is

[TABLE]

with $\Delta_{\min}$ and $\Delta_{\max}$ as in (68) and (69), respectively. It suffices if $|{\mathcal{U}}|\leq 2|\Theta|+1.$

Proof: See Appendix A-B.

The achievability proof of Theorem 3, by dint of Proposition 4, will use a deterministic sampler based on the minimizing $w$ from (71) or (72).

Example 6**.**

This example compares the USRDfs for a $k$-MRS and a $k$-IRS and is an adaptation of Example 2 above (and also of ([3], Example 2)). Consider Example 2 with $q_{\tau}=0.5$ for every $\tau\in\Theta$, whereby $P_{X_{1}X_{2}|\theta=\tau}=P_{X_{1}|\theta=\tau}P_{X_{2}|\theta=\tau}$. Clearly, $\Theta_{2}=\Theta$. For a $k$-IRS, the Bayesian USRDf is

[TABLE]

for $0\leq\Delta\leq p$, where $p=\mathbbm{E}[p_{\theta}]$, and the nonBayesian USRDf is

[TABLE]

For a $k$-MRS, in $\rho_{m}^{{\mathcal{B}}}(\delta,P_{U},P_{S|X_{{\mathcal{M}}}U},\tau)$ as well as $\rho_{m}^{n{\mathcal{B}}}(\delta,P_{U},P_{S|X_{{\mathcal{M}}}U},\tau)$, $P_{U}=$ a point-mass and

[TABLE]

are uniformly optimal for all $0\leq\delta\leq p_{\tau}$ and for all $\tau\in\Theta$. Then, the Bayesian USRDf is

[TABLE]

and the nonBayesian USRDf is

[TABLE]

Clearly, in both the Bayesian and nonBayesian settings $R_{m}(\Delta)<R_{\imath}(\Delta)$.

∎

In closing this section, standard properties of the USRDf for the fixed-set sampler, $k$-IRS and $k$-MRS in the Bayesian and nonBayesian settings are summarized below, with the proof provided in Appendix A-C.

Lemma 5**.**

The right-sides of (16), (19), (47), (49), (65) and (67) are finite-valued, decreasing, convex, continuous functions of $\Delta_{\min}\leq\Delta\leq\Delta_{\max}$.

IV Proofs

IV-A Achievability proofs

Our achievability proofs emphasize the Bayesian setting. Counterpart proofs in the nonBayesian setting use similar sets of ideas, and so we limit ourselves to pointing out only the distinctions between these and their Bayesian brethren. In the Bayesian setting, the achievability proofs successively build upon each other according to increasing complexity of the sampler, and are presented in the order: fixed-set sampler, $k$-IRS and $k$-MRS.

A common theme in the achievability proofs for a $k$-FS, a $k$-IRS and a $k$-MRS involves forming estimates ${\widehat{\tau}_{1}}$ of the underlying $\tau_{1}$ in $\Theta_{1}$, ${\widehat{\tau}_{2}}$ of $\tau_{2}$ in $\Theta_{2}$ and ${\widehat{\tau}}$ of $\tau$ in $\Theta$, respectively. The assumed finiteness of $\Theta$ enables ${\widehat{\tau}_{1}}$ or ${\widehat{\tau}_{2}}$ to be conveyed rate-free to the decoder. Codes for achieving USRDf at a prescribed distortion level $\Delta$ are chosen from among fixed-set sampling rate distortion codes for $\tau_{1}$s in $\Theta_{1}$ or from among IRS codes for $\tau_{2}$s in $\Theta_{2}$ or from among MRS codes for $\tau$s in $\Theta$. Such codes, in the Bayesian setting, correspond to appropriate distortion thresholds that, in effect, average to yield a distortion level $\Delta$; in the nonBayesian setting, a suitable “worst-case” distortion must not exceed $\Delta$. A chosen code corresponds to an estimate ${\widehat{\tau}_{1}}$, ${\widehat{\tau}_{2}}$ or ${\widehat{\tau}}$.

A mainstay of our achievability proofs is the existence of sampling rate distortion codes with fixed-set sampling for a DMMS with known pmf $Q$.

Lemma 6**.**

Consider a DMMS $\{X_{{\mathcal{M}}t}\}_{t=1}^{\infty}$ with known pmf $Q=Q_{X_{{\mathcal{M}}}}$. Let $A,B\subseteq{\mathcal{M}}$ be fixed sampling and recovery sets, respectively, and define

[TABLE]

For every $\epsilon>0$ and $\Delta_{\min}\leq\Delta\leq\Delta_{\max}$, there exists a sampling rate distortion code $(f,\varphi)$ of rate

[TABLE]

and expected distortion

[TABLE]

for all $n$ large enough. Here,

[TABLE]

Proof: The proof of the lemma follows from the achievability proof of Proposition 1 in [3] upon replacing the recovery set ${\mathcal{M}}$ therein by $B$. ∎

Theorem 1: Considering first the Bayesian setting, observe that

[TABLE]

and

[TABLE]

Now, consider a partition $\Theta_{1}$ of $\Theta$ as in Section III. Based on the sampler output $X_{A}^{n},$ the encoder forms an ML estimate of $\theta_{1}$ as

[TABLE]

For each $\tau_{1}$ in $\Theta_{1}$, observe that $\{X_{At}\}_{t=1}^{\infty}$ is a DMMS with pmf $P_{\tau_{1}}\triangleq P_{X_{A}|\theta_{1}=\tau_{1}}$. The sequence of ML estimates $\{{\widehat{\tau}}_{1,n}\}_{n}$ converges in $P_{\tau_{1}}$-probability to $\tau_{1}$, so that for every $\epsilon>0$ and $\tau_{1}$ in $\Theta_{1}$, there exists an $N_{1}(\epsilon,\tau_{1})$ such that

[TABLE]

where $d_{\max}=\underset{x_{B}\in{\mathcal{X}}_{B},\ y_{B}\in{\mathcal{Y}}_{B}}{\max}\ d(x_{B},y_{B})$. By the finiteness of $\Theta_{1}$, there exists an $N(\epsilon)$ such that simultaneously for all $\tau_{1}\in\Theta_{1}$,

[TABLE]

and consequently

[TABLE]

For a fixed $\Delta_{\min}\leq\Delta\leq\Delta_{\max}$, let $\{\Delta_{\tau_{1}},\ \tau_{1}\in\Theta_{1}\}$ yield the minimum in (16). For each $\tau_{1}$ in $\Theta_{1}$, for the DMMS $\{X_{{\mathcal{M}}t}\}_{t=1}^{\infty}$ with pmf $P_{X_{{\mathcal{M}}}|\theta_{1}=\tau_{1}}$ and distortion measure $d_{\tau_{1}}$, there exists by Lemma 6 – with $Q=P_{X_{{\mathcal{M}}}|\theta_{1}=\tau_{1}}$ and $d_{A}=d_{\tau_{1}}$ – a fixed-set sampling rate distortion code $(f_{\tau_{1}},\varphi_{\tau_{1}}),$ $f_{\tau_{1}}:{\mathcal{X}}_{A}^{n}\rightarrow\{1,\ldots,J\}$ and $\varphi_{\tau_{1}}:\{1,\ldots,J\}\rightarrow{\mathcal{Y}}_{B}^{n}$ of rate $\frac{1}{n}\log J\leq\underset{\tau_{1}\in\Theta_{1}}{\max}\ \rho_{A}^{{\mathcal{B}}}(\Delta_{{\tau_{1}}},{\tau_{1}})+\frac{\epsilon}{2}=R_{A}(\Delta)+\frac{\epsilon}{2}$ and with expected distortion

[TABLE]

for all $n\geq N_{2}(\epsilon,\tau_{1})$.

A code $(f,\varphi)$, with $f$ taking values in ${\mathcal{J}}\triangleq\{1,\ldots,|\Theta_{1}|\}\times\{1,\ldots,J\}$ is constructed as follows. Order (in any manner) the elements of $\Theta_{1}$. The encoder $f$, dictated by the estimate ${\widehat{\tau}}_{1,n}$, is

[TABLE]

The decoder is

[TABLE]

The rate of the code is

[TABLE]

for all $n$ large enough, by the finiteness of $\Theta_{1}$.

The code $(f,\varphi)$ is seen to satisfy

[TABLE]

The first term on the right-side of (95) is

[TABLE]

Combining (95) and (101),

[TABLE]

by (90) for all $n$ large enough. Finally, we note that (92) and (103) hold simultaneously for all $n$ large enough.

In the nonBayesian setting, the achievability proof follows by adapting the steps above with the following differences. For each $\tau_{1}$ in $\Theta_{1}$, a fixed-set sampling rate distortion code $(f_{\tau_{1}},\varphi_{\tau_{1}})$ is chosen now with expected distortion $\mathbbm{E}[d(X_{B}^{n},\varphi_{\tau_{1}}(f_{\tau_{1}}(X_{A}^{n})))|\theta=\tau]\leq\Delta+\frac{\epsilon}{2}$ for every $\tau$ in $\Lambda(\tau_{1})$ and of rate $\frac{1}{n}\log||f_{\tau_{1}}||\leq R_{A}(\Delta)+\frac{\epsilon}{2}$, where $R_{A}(\Delta)$ is the nonBayesian USRDf for a fixed-set sampler.

∎

Theorem 2: In the Bayesian setting, for a given $\Delta_{\min}\leq\Delta\leq\Delta_{\max},$ consider the $P_{S},\{\Delta_{\tau_{2}},\ \tau_{2}\in\Theta_{2}\}$ that attain the (outer) minimum in (47). For the corresponding minimizing $P_{Y_{B}|SX_{S}\theta_{2}}$ in (47) (by way of (42))

[TABLE]

and let

[TABLE]

The second expression in (104) suggests an achievability scheme using an IRS code (see [3]) governed by $\theta_{2}$. Our achievability proof comprises two phases. In the first phase an estimate ${\widehat{\tau}_{2}}$ of $\theta_{2}$ is formed based on the output of a $k$-IRS that chooses each $A_{i}$ in ${\mathcal{A}}_{k}$ repeatedly for $N$ time instants. The second phase, of length $n$, entails choosing each $S_{t}=A_{i}$ repeatedly for $\approx nP_{S}(A_{i})$ time instants and an IRS code governed by ${\widehat{\tau}_{2}}$ of expected distortion

[TABLE]

is applied to the output of the sampler. This predetermined selection of sampling sets obviates the need for the decoder to be additionally informed.

Denote $|{\mathcal{A}}_{k}|$ by $M_{k}={m\choose k}.$ Fix $\epsilon>0$ and $0<\epsilon^{\prime}<\epsilon$. In the first phase, a $k$-IRS is chosen to sample each $A_{i}\in{\mathcal{A}}_{k}$ over disjoint time-sets $\mu_{i}$ of length $N$. The union of the time-sets $\mu_{i},\ i\in{\mathcal{M}}_{k}\triangleq\{1,\ldots,M_{k}\}$ is denoted by $\mu\triangleq\{1,\ldots,M_{k}N\}$. Based on the sampler output, an ML estimate ${\widehat{\tau}}_{2,N}={\widehat{\tau}}_{2,N}(S^{\mu},X_{S}^{\mu})$ of $\theta_{2}$ is formed with

[TABLE]

for $N\geq N_{\epsilon^{\prime}}$, say.

In the second phase, we denote the next set of $n$ time instants, i.e., $\{M_{k}N+1,\ldots,M_{k}N+n\}$ simply by $\nu\triangleq\{1,\ldots,n\}.$ Further, for each $i$ in ${\mathcal{M}}_{k}$, define the time-sets $\nu_{A_{i}}\subset\nu$, made up of consecutive time instants, as

[TABLE]

and note that the union of $\nu_{A_{i}}$s is $\nu$, and

[TABLE]

In this phase, the $k$-IRS is now chosen (deterministically) as follows:

[TABLE]

For each DMMS $\{X_{{\mathcal{M}}t}\}_{t=1}^{\infty}$ with pmf $P_{X_{{\mathcal{M}}}|\theta_{2}=\tau_{2}},\ \tau_{2}\in\Theta_{2}$, and for each $A_{i}$ in ${\mathcal{A}}_{k}$ and its corresponding distortion measure $d_{\tau_{2}},$ there exists by Lemma 6 – with $Q=P_{X_{{\mathcal{M}}}|\theta_{2}=\tau_{2}}$ and $d_{A}=d_{\tau_{2}}$ – a fixed-set sampling rate distortion code $(f_{A_{i}}^{\tau_{2}},\varphi_{A_{i}}^{\tau_{2}}),\ f_{A_{i}}^{\tau_{2}}:{\mathcal{X}}_{A_{i}}^{\nu_{A_{i}}}\rightarrow\{1,\ldots,J_{A_{i}}^{\tau_{2}}\}$ and $\varphi_{A_{i}}^{\tau_{2}}:\{1,\ldots,J_{A_{i}}^{\tau_{2}}\}\rightarrow{\mathcal{Y}}_{B}^{\nu_{A_{i}}}$ of rate $\frac{1}{|\nu_{A_{i}}|}\log J_{A_{i}}^{\tau_{2}}\leq I(X_{A_{i}}\wedge Y_{B}|S=A_{i},\theta_{2}={\tau_{2}})+\frac{\epsilon^{\prime}}{4}$ (cf. (104)) and with

[TABLE]

for all $|\nu_{A_{i}}|\geq N_{A_{i}}({\epsilon^{\prime}},\tau_{2}).$ Note that

[TABLE]

and

[TABLE]

for every $\tau_{2}$ in $\Theta_{2}$.

Consider a (composite) code $(f,\varphi)$ as follows. Denote $n^{\prime}\triangleq|\mu|+|\nu|=M_{k}N+n,$ and the encoder $f$ consisting of a concatenation of encoders is defined by

[TABLE]

The decoder $\varphi$, which is aware of the predetermined sequence of sampling sets, is defined by

[TABLE]

for each encoder output $({\widehat{{\tau}}}_{2,N},j_{1},\ldots,j_{M_{k}})$, where $y_{1}\in{\mathcal{Y}}_{\mathcal{M}}$ is an arbitrary symbol. Clearly, $|\Theta_{2}|\times\underset{\tau_{2}\in\Theta_{2}}{\max}\prod\limits_{i=1}^{M_{k}}J_{A_{i}}^{{\tau_{2}}}$ indices would suffice to describe all possible encoder outputs.

The rate of the code is

[TABLE]

where the previous inequality holds for all $n$ large enough. Denoting the output of the decoder by $Y_{B}^{n^{\prime}}\triangleq\varphi(f(S^{n^{\prime}},X_{S}^{n^{\prime}}))$

[TABLE]

The first two terms on the right-side of (115) are

[TABLE]

by (107) for $N$ large enough, and the last term on the right-side of (115) is

[TABLE]

From (115)-(121), we have

[TABLE]

for $n$ and $N$ large enough. Finally, we note that (114) and (122) hold simultaneously for all $n$ and $N$ large enough.

The Corollary is immediate by the choice of codes with “uninformed” decoder in the proof above.

For the nonBayesian setting, achievability follows by adapting the proof above in a manner similar to that for a $k$-FS in Theorem 1.

∎

Theorem 3: The achievability proof relies on the deterministic sampler justified by Proposition 4. In the Bayesian setting, for a given $\Delta_{\min}\leq\Delta\leq\Delta_{\max}$, let $P_{U},\ P_{S|X_{{\mathcal{M}}}U}=\delta_{w},\ \{\Delta_{\tau},\ \tau\in\Theta\}$ attain the minimum in (71). For the corresponding minimizing $P_{Y_{B}|SX_{S}U\theta}$ in (61), the right-side of (71) is

[TABLE]

and we set

[TABLE]

Our achievability proof uses a $k$-MRS in two distinct modes. First, a deterministic $k$-MRS is chosen so as to form an estimate ${\widehat{\tau}}$ of $\theta$ from the sampler output. Next, for each $U=u$, a suitable deterministic $k$-MRS is chosen in accordance with $w(x_{{\mathcal{M}}},u)$, and an MRS code (see [3]) governed by ${\widehat{\tau}}$ of expected distortion

[TABLE]

is applied to the sampler output. Concatenation of such codes corresponding to various $u\in{\mathcal{U}}$ yields, in effect, time-sharing that serves to achieve (123). To simplify the notation, the conditioning on $U=u$ will be suppressed except when needed.

Fix $\epsilon>0$ and $0<\epsilon^{\prime}<\epsilon$.

(i) We devise a deterministic $k$-MRS on a time-set $\mu$, based on whose output an estimate ${\widehat{\tau}}_{N}={\widehat{\tau}}_{N}(S^{\mu},X_{S}^{\mu})={\widehat{\tau}}_{N}(S^{\mu})$ of $\theta$ is formed with

[TABLE]

for $N\geq N_{\epsilon^{\prime}}$. The estimate ${\widehat{\tau}}_{N}$ is formed from only the sampling sequence $S^{\mu}$ and thus is available to the encoder as well as the decoder. The $k$-MRS is chosen on the time-set $\mu$, to signal the occurrences of each $x\in{\mathcal{X}}_{\mathcal{M}}$ to the encoder and decoder through $S^{\mu}$ above; for each $x\in{\mathcal{X}}_{\mathcal{M}},$ a distinct $A\in{\mathcal{A}}_{k}$ is chosen. If $|{\mathcal{A}}_{k}|\geq|{\mathcal{X}}_{\mathcal{M}}|$, a trivial one-to-one mapping from ${\mathcal{X}}_{\mathcal{M}}$ to ${\mathcal{A}}_{k}$ enables $S^{\mu}$ to determine $X_{{\mathcal{M}}}^{\mu}$, where $S^{\mu}$ is of length $N$, say. Then ${\widehat{\tau}}_{N}$ is taken to be the ML estimate of $\theta$ based on $X_{{\mathcal{M}}}^{\mu}$, which satisfies (125).

When $|{\mathcal{A}}_{k}|<|{\mathcal{X}}_{\mathcal{M}}|,$ a $k$-MRS is chosen attuned variously to disjoint subsets of ${\mathcal{X}}_{\mathcal{M}}$, of size $|{\mathcal{A}}_{k}|-1$, on corresponding disjoint time-sets $\mu_{l}$ of length $N$, $l=1,\ldots,\left\lceil\frac{|{\mathcal{X}}_{\mathcal{M}}|}{|{\mathcal{A}}_{k}|-1}\right\rceil$, as follows. In each $\mu_{l}$, the $k$-MRS signals the occurrence (or not) of $X_{{\mathcal{M}}t}=x$ in the $l^{th}$-subset of ${\mathcal{X}}_{\mathcal{M}}$ in a (deterministic) manner by choosing $|{\mathcal{A}}_{k}|-1$ distinct sampling sets in ${\mathcal{A}}_{k}$; the nonoccurrence of symbols from this $l^{th}$-subset of ${\mathcal{X}}_{\mathcal{M}}$ is indicated by the remaining (dummy) sampling set in ${\mathcal{A}}_{k}$. We denote $\bigcup\limits_{l}\mu_{l}$ by $\mu$. Finally, ${\widehat{\tau}}_{N}$ is taken as the ML estimate of $\theta$ based on the sampling sequence $S^{\mu}$ of length $\left\lceil\frac{|{\mathcal{X}}_{\mathcal{M}}|}{|{\mathcal{A}}_{k}|-1}\right\rceil N=N^{\prime}$, say.

(ii) Next, for each $U=u$, a $k$-MRS is chosen according to $P_{S|X_{{\mathcal{M}}},U=u}=\delta_{w(\cdot,u)}$ for $n$ time instants. Then, for a DMMS $\{X_{{\mathcal{M}}t}\}_{t=1}^{\infty}$ with pmf $P_{X_{{\mathcal{M}}}|\theta={\widehat{\tau}}_{N}}$ an MRS code comprising a concatenation of fixed-set sampling rate distortion codes corresponding to the $A_{i}$s in ${\mathcal{A}}_{k}$ is applied to the sampler output.

Denote the set of $n$ time instants $\{N^{\prime}+1,\ldots,N^{\prime}+n\}$ simply by $\gamma\triangleq\{1,\ldots,n\}.$ Define time-sets ${\gamma_{S^{n}}(A_{i})}\triangleq\{t:1\leq t\leq n,\ S_{t}=A_{i}\},\ i\in{\mathcal{M}}_{k},$ and note that ${\gamma_{S^{n}}(A_{i})}$s cover $\gamma$, i.e.,

[TABLE]

Denote the set of the first $\max\{\lceil(nP_{S|\theta}(A_{i}|{\widehat{\tau}}_{N}))-\epsilon^{\prime}\rceil,0\}$ time instants in each $\gamma_{S^{n}}(A_{i})$ by $\nu_{A_{i}}$ (suppressing the dependence on ${\widehat{\tau}}_{N}$). Defining the (typical) set for each $\tau$ in $\Theta$

[TABLE]

we have that

[TABLE]

for all $n$ large enough.

By Lemma 6, for each DMMS $\{X_{{\mathcal{M}}t}\}_{t=1}^{\infty}$ with pmf $P_{X_{{\mathcal{M}}}|S=A_{i},\theta=\tau},\ i\in{\mathcal{M}}_{k},\ \tau\in\Theta,$ there exists a code $(f_{A_{i}}^{\tau},\varphi_{A_{i}}^{\tau}),\ f_{A_{i}}^{\tau}:{\mathcal{X}}_{A_{i}}^{\nu_{A_{i}}}\rightarrow\{1,\ldots,J_{A_{i}}^{\tau}\}$ and $\varphi_{A_{i}}^{\tau}:\{1,\ldots,J_{A_{i}}^{\tau}\}\rightarrow{\mathcal{Y}}_{B}^{\nu_{A_{i}}}$ of rate

[TABLE]

and with

[TABLE]

for all $|\nu_{A_{i}}|\geq N_{A_{i}}({\epsilon^{\prime}},\tau)$. Such codes are considered for each $U=u.$

Consider a (composite) code $(f,\varphi)$ as follows. Denoting $N^{\prime}+n$ by $n^{\prime}$, an encoder $f$ consisting of a concatenation of encoders is defined as

[TABLE]

For $t=1,\ldots,n^{\prime},$ and each encoder output $(j_{1},\ldots,j_{M_{k}})$, the decoder $\varphi$, which can recover the estimate ${\widehat{\tau}}_{N}$ from its knowledge of the sampling sequence $S^{n^{\prime}}=s^{n^{\prime}}$, is given by

[TABLE]

where $y_{1}$ is a fixed but arbitrary symbol in ${\mathcal{Y}}_{\mathcal{M}}.$

Finally, for $N$ and $n$ large enough, the codes $(f,\varphi)$ corresponding to each $U=u$ are concatenated so as to effect the time-sharing prescribed by $P_{U}$, in a standard manner. It is shown in Appendix A-A that the rate of the resulting code is

[TABLE]

using (129) and the expected distortion is

[TABLE]

from (125), (128), (130) and the definition of $\Delta_{A_{i},u,\tau}$. ∎

IV-B Converse proof

In contrast with the achievability proofs, we present a unified converse proof for Theorems 3, 2 and 1 according to successive weakening of the sampler, viz. $k$-MRS, $k$-IRS and fixed-set sampler. We begin with the technical Lemma 7 that is used subsequently in the converse proof.

Lemma 7**.**

Let finite-valued rvs $C,D^{n},E^{n},F^{n},$ be such that $(D_{t},E_{t}),\ t=1,\ldots,n,$ are conditionally mutually independent given $C$, i.e.,

[TABLE]

and satisfy

[TABLE]

For any function $g(C)$ of $C$, such that

[TABLE]

it holds that

[TABLE]

Proof: First, from (138), we have

[TABLE]

Now, the second term on the right-side of (143) is

[TABLE]

Next, the first part of (139) along with (143) implies that

[TABLE]

and hence

[TABLE]

Now, by (147) and (150), for $t=1,\ldots,n,$

[TABLE]

which is the claim (140).

∎

Converse: In the Bayesian setting, we provide first a converse proof for Theorem 3, which is then refashioned to give converse proofs for Theorems 2 and 1.

Let $(\{P_{S_{t}|X_{{\mathcal{M}}t}\theta}=P_{S_{t}|X_{{\mathcal{M}}t}}\}_{t=1}^{\infty},f,\varphi)$ be an $n$-length $k$-MRS block code of rate $R$ and with decoder output $Y_{B}^{n}=\varphi(S^{n},f(S^{n},X_{S}^{n}))$ satisfying $\mathbbm{E}[d(X_{B}^{n},Y_{B}^{n})]\leq\Delta.$ The hypothesis of Lemma 7 is met with $C=\theta,\ D^{n}=X_{{\mathcal{M}}}^{n},\ E^{n}=(S^{n},X_{S}^{n}),\ F^{n}=Y_{B}^{n}$ and $g(\theta)=\theta$, since

[TABLE]

while

[TABLE]

holds by code construction. Also, (152) implies, upon summing over all realizations of $X_{S^{c}}^{n}$, that

[TABLE]

Then the claim of the lemma implies that

[TABLE]

Let $\Delta_{\tau}$ denote $\mathbbm{E}[d(X_{B}^{n},Y_{B}^{n})|\theta=\tau]=\frac{1}{n}\sum\limits_{t=1}^{n}\mathbbm{E}[d(X_{Bt},Y_{Bt})|\theta=\tau]$ for each $\tau$ in $\Theta$ and note that $\mathbbm{E}[\Delta_{\theta}]\leq\Delta$. For every $\tau$ in $\Theta$, the following holds:

[TABLE]

By (155),

[TABLE]

lies in the convex hull of

[TABLE]

By the Carathéodory Theorem [4], every point in the convex hull of ${\cal C}$ can be represented as a convex combination of at most $2|\Theta|+1$ elements in ${\cal C}$. The corresponding pmfs are indexed by the values of a rv $U$ with

[TABLE]

where the pmf of $U$ has support of size $\leq 2|\Theta|+1$. Then, in a standard manner, (162) leads to

[TABLE]

Now, (167) holds for every $\tau\in\Theta$, and hence

[TABLE]

for $\Delta\geq\Delta_{\min}.$

Turning next to Theorems 2 and 1, an $n$-length $k$-IRS code or a fixed-set sampling block code can be viewed as restrictions of a $k$-MRS code. Specifically, in Theorem 2, for a $k$-IRS code of rate $R$ with $P_{S_{t}},\ g(\theta)=\theta_{2}$ instead of $P_{S_{t}|X_{{\mathcal{M}}t}},\ g(\theta)=\theta$ (for a $k$-MRS), the hypothesis of Lemma 7 holds. Denote $\mathbbm{E}[d(X_{B}^{n},Y_{B}^{n})|\theta_{2}=\tau_{2}]$ by $\Delta_{\tau_{2}},\ \tau_{2}\in\Theta_{2}$. Then, the pmfs in (165) satisfy

[TABLE]

The counterpart of (166) is

[TABLE]

noting from (171) that $P_{U|S,\theta_{2}}=P_{U|S}.$ Using the convexity of the mutual information terms above with respect to $P_{Y_{B}|SX_{S}U\theta_{2}}$, we get

[TABLE]

Since (175) holds for every $\tau_{2}\in\Theta_{2}$

[TABLE]

i.e., $R\geq R_{\imath}(\Delta),\ \Delta\geq\Delta_{\min},$ completing the converse proof of Theorem 2.

In a manner analogous to a $k$-IRS, in Theorem 1 for a fixed-set sampler the hypothesis of Lemma 7 holds with $P_{S_{t}}=\mathbbm{1}(S_{t}=A),\ g(\theta)=\theta_{1}$. Defining $\Delta_{\tau_{1}}\triangleq\mathbbm{E}[d(X_{B}^{n},Y_{B}^{n})|\theta_{1}=\tau_{1}],\ \tau_{1}\in\Theta_{1}$, the counterpart of the right-side of (168) reduces to $\underset{\tau_{1}\in\Theta_{1}}{\max}\ \rho_{A}^{{\mathcal{B}}}(\Delta_{\tau_{1}},\tau_{1}).$ It then follows that

[TABLE]

providing the converse proof for Theorem 1.

In the nonBayesian setting, the analog of Lemma 7 is obtained similarly with $C=c,\ g(C)=g(c),$ and (137)–(140) expressed in terms of appropriate conditional pmfs. The converse proofs for a $k$-MRS, $k$-IRS and $k$-FS are obtained as above but by excluding the outer minimizations over $\{\Delta_{\tau},\tau\in\Theta\}$, $\{\Delta_{\tau_{2}},\tau_{2}\in\Theta_{2}\}$ and $\{\Delta_{\tau_{1}},\tau_{1}\in\Theta_{1}\}$, respectively.

∎

V Discussion

Our formulation of universality requires optimum sampling rate distortion performance when the “true” underlying pmf of the DMMS belongs to a finite family ${\mathcal{P}}=\{P_{X_{{\mathcal{M}}}|\theta=\tau},\ \tau\in\Theta\}$. The assumed finiteness of $\Theta$ affords two benefits in addition to mathematical ease: (i) simple proofs of estimator consistency uniformly over $\Theta_{1},\ \Theta_{2}$ or $\Theta$; and (ii) rate-free conveyance of corresponding estimates ${\widehat{\tau}}_{1},\ {\widehat{\tau}}_{2}$ or ${\widehat{\tau}}$ to the decoder. General extensions to the case when $\Theta$ is an infinite set (countable or uncountable) remain open.

Unlike for a $k$-IRS, the assumption in a $k$-MRS that the decoder is informed of the sampling sequence $S^{n}$ plays an important role. Specifically, embedded information regarding $X_{{\mathcal{M}}}^{n}$ is conveyed implicitly to the decoder through $S^{n}$. Also, as a side-benefit, the decoder can replicate the estimate of $\theta$ formed by the encoder based on $S^{n}$ alone, obviating the need for explicitly transmitting it. However, if the decoder were denied a knowledge of $S^{n}$, what is the USRDf? This question, too, remains unanswered.

Underlying our achievability proofs of Theorems 2 and 3 for a $k$-IRS and $k$-MRS, are schemes for distribution-estimation based on $(S^{n},X_{S}^{n}).$ A distinguishing feature from classical estimation settings is the additional degree of (spatial) freedom in the choice of the sampling sequence $S^{n}$. This motivates questions of the following genre: How should $S^{n}$, consisting of (possibly different) $k$-sized subsets, be chosen to form “best” estimates of the underlying joint pmf? How does the degree of the allowed dependence of $S^{n}$ on $X_{{\mathcal{M}}}^{n}$ affect estimator performance? For instance, our choice of sampling sequence and estimation procedure in the achievability proof of Theorem 3 is a simple starting point. How must we devise efficient sampling mechanisms to exploit an implicit embedding of DMMS realization in the sampler output? These questions are of independent interest in statistical learning theory.

Appendix A Appendices

A-A Proof of (134) and (136)

For the code formed by concatenating $(f,\varphi)$ for each $u\in{\mathcal{U}}$, the rate is

[TABLE]

for all $n$ large enough.

For each $U=u,$ let $\Delta_{u}\triangleq\sum\limits_{\tau\in\Theta,\ A_{i}\in{\mathcal{A}}_{k}}\mu_{\theta}(\tau)P_{S|U\theta}(A_{i}|u,\tau)\Delta_{A_{i},u,\tau}$. Denoting the output of the decoder by $Y_{B}^{n^{\prime}},$ we get

[TABLE]

for all $n,N$ large enough, where the previous inequality is shown below. Then, expected distortion for the code formed by concatenating $(f,\varphi)$ for each $u\in{\mathcal{U}}$, is

[TABLE]

It remains to show (187). Now, (187) follows from the following: In (186), for each $\tau\in\Theta$ and $s^{n}\in{\cal T}^{(n)}({\epsilon^{\prime}},{\widehat{\tau}}_{N})$,

[TABLE]

for all $n$ large enough and $\epsilon^{\prime}$ chosen appropriately. ∎

A-B *Proof of Proposition 4 *

First, for the Bayesian setting, by Theorem 3, the claim entails showing that

[TABLE]

for $\Delta_{\min}\leq\Delta\leq\Delta_{\max}$. Denote the expressions in (197) and (198) by $q(\Delta)$ and $r(\Delta),$ respectively. Now, from the conditional version of Topsøe’s identity [5, Lemma 8.5], observe that $q(\Delta)$ equals

[TABLE]

Note that the inner max and min can be interchanged in (199). Denoting D\left(P_{Y_{B}|SX_{S}U,\theta=\tau}\big{|}\big{|}Q_{Y_{B}|SU,\theta=\tau}\big{|}P_{SX_{S}U|\theta=\tau}\right) by $D_{\tau},\ \tau\in\Theta$, we write (199) as

[TABLE]

which is the epigraph form. Also, $r(\Delta)$ can be expressed in a similar manner. Based on (202), we define $G_{q}(\alpha,\{\lambda_{\tau},\tau\in\Theta\})$ and $G_{r}(\alpha,\{\lambda_{\tau},\tau\in\Theta\})$ in terms of the Lagrangians of $q(\Delta)$ and $r(\Delta)$, respectively, in a standard way.

Specifically, $G_{q}(\alpha,\{\lambda_{\tau},\ \tau\in\Theta\})$

[TABLE]

Let $P_{\tau}\triangleq P_{X_{{\mathcal{M}}}|\theta=\tau}$. When $\sum\limits_{\tau\in\Theta}\lambda_{\tau}=1,$ from (205), $G_{q}(\alpha,\{\lambda_{\tau},\ \tau\in\Theta\})$ equals

[TABLE]

where the expectation above is with respect to $P_{Y_{B}|S=s,X_{S}=x_{s},U=u,\theta=\tau}$. Noting that the term \Big{(}\cdots\Big{)} above is a function of $s,x_{{\mathcal{M}}},u,$ we get

[TABLE]

Since $q(\Delta)$ and $r(\Delta)$ are convex in $\Delta,$ they can be expressed in terms of their respective Lagrangians as

[TABLE]

Thus,

[TABLE]

upon observing that the maxima in (215) are attained when $\sum\limits_{\tau\in\Theta}\lambda_{\tau}=1$.

∎

A-C * Proof of Lemma 5*

Clearly, for each $\tau_{1}\in\Theta_{1}$, $\rho_{A}^{{\mathcal{B}}}(\delta,\tau_{1})$ and $\rho_{A}^{n{\mathcal{B}}}(\delta,\tau_{1})$ are finite-valued and, hence, so are the right-sides of (16) and (19). Also, they are also nonincreasing in $\Delta$. The convexity of the right-sides of (16) and (19) follows from the convexity of $\rho_{A}^{{\mathcal{B}}}(\delta,\tau_{1})$ and $\rho_{A}^{n{\mathcal{B}}}(\delta,\tau_{1})$ in $\delta$ along with a standard argument shown below; continuity for $\Delta>\Delta_{\min}$ is a consequence. Continuity at $\Delta_{\min}$ holds, for instance, as in ([5], Lemma 7.2). The claimed properties of the right-sides of (47), (49), (65) and (67) follow in a similar manner.

The convexity of the right-side of (16) can be shown explicitly as follows. Let $\tau_{1}(1)$ and $\tau_{1}(2)$ attain the maximum in (16) at $\Delta=\Delta_{1}$ and $\Delta=\Delta_{2}$, respectively, where $\Delta_{1}<\Delta_{2}$. The corresponding minimizing $\{\Delta_{\tau_{1}},\tau_{1}\in\Theta_{1}\}$ are denoted by $\{\Delta_{\tau_{1}}^{1},\ \tau_{1}\in\Theta_{1}\}$ and $\{\Delta_{\tau_{1}}^{2},\ \tau_{1}\in\Theta_{1}\}$, respectively. For any $0<\alpha<1$, for $i=1,\ldots,|\Theta_{1}|$

[TABLE]

where the inequality above follows by Remark (iii) preceding Theorem 1 in Section III. Now, (221) holds for every $i=1,\ldots,|\Theta_{1}|,$ hence

[TABLE]

∎