Communication Complexity of Estimating Correlations

TL;DR

This paper determines the optimal communication complexity for estimating correlation coefficients between distributed Gaussian or binary variables, revealing that a simple one-way protocol achieves the best possible mean squared error rate.

Contribution

It establishes the exact asymptotic communication cost for correlation estimation and introduces a one-way protocol that attains this optimal performance.

Findings

Optimal $rac{1}{k}$ mean squared error rate with $k$ bits exchanged.

One-way protocol suffices to achieve optimal performance.

Lower bounds hold even for small ranges of $ ho$, indicating tight bounds.

Abstract

We characterize the communication complexity of the following distributed estimation problem. Alice and Bob observe infinitely many iid copies of $\rho$-correlated unit-variance (Gaussian or $\pm1$ binary) random variables, with unknown $\rho\in[-1,1]$. By interactively exchanging $k$ bits, Bob wants to produce an estimate $\hat\rho$ of $\rho$. We show that the best possible performance (optimized over interaction protocol $\Pi$ and estimator $\hat \rho$) satisfies $\inf_{\Pi \hat\rho}\sup_\rho \mathbb{E} [|\rho-\hat\rho|^2] = \tfrac{1}{k} (\frac{1}{2 \ln 2} + o(1))$. Curiously, the number of samples in our achievability scheme is exponential in $k$; by contrast, a naive scheme exchanging $k$ samples achieves the same $\Omega(1/k)$ rate but with a suboptimal prefactor. Our protocol achieving optimal performance is one-way (non-interactive). We also prove the $\Omega(1/k)$ bound even when $\rho$ is restricted to any small open sub-interval of $[-1,1]$ (i.e. a local minimax lower bound). Our proof techniques rely on symmetric strong data-processing inequalities and various tensorization techniques from information-theoretic interactive common-randomness extraction. Our results also imply an $\Omega(n)$ lower bound on the information complexity of the Gap-Hamming problem, for which we show a direct information-theoretic proof.

Equations

$$f(\mathbf{X},\mathbf{Y})=\underset{n\to \infty }{\lim }\frac{1}{n}\sum _{i=1}^n{X}_i{Y}_i$$

$$\displaystyle\begin{array}[]{cc}\textbf{mutual information {\em interchanged} between Alice and Bob}&\\ \leq&\\ \rho^{2}\times\textbf{mutual information {\em injected} by Alice and Bob}&\end{array}$$

$$R_{\rho }(\Pi ,\hat{\rho })\triangleq E_{\rho }{\left(\hat{\rho }(\Pi ,\mathbf{Y})-\rho \right)}^2,$$

$$R^{\ast }\triangleq \underset{n,\Pi ,\hat{\rho }}{\inf }\underset{-1\le \rho \le 1}{\sup }{R}_{\rho }(\Pi ,\hat{\rho }),$$

$$R_{\rho ,\delta }^{\ast }\triangleq \underset{n,\Pi ,\hat{\rho }}{\inf }\underset{|{\rho }^{\prime }-\rho |\le \delta }{\sup }{R}_{{\rho }^{\prime }}(\Pi ,\hat{\rho }).$$

$$R_{\rho ,\delta }^{\ast }\le \frac{1}{k}\left(\frac{{\left(1-{\rho }^2\right)}^2}{2\ln 2}+o(1)\right),$$

$$R^{\ast }\le \frac{1}{k}\left(\frac{1}{2\ln 2}+o(1)\right).$$

$$R_{\rho ,\delta }^{\ast }\ge \frac{(1-|\rho |{)}^2}{2k\ln 2}(1+o(1)).$$

$$R^{\ast }\ge \frac{1}{k}\left(\frac{1}{2\ln 2}+o(1)\right).$$

$$\mathrm{Var}\hat{\rho }\ge \frac{(1-|\rho |{)}^2}{2k\ln 2}.$$

$$D(P_{\mathbf{X}\Pi }^{\rho }\Vert P_{\mathbf{X}\Pi }^0)\le {\rho }^{2}I(\Pi ;\mathbf{X},\mathbf{Y}).$$

$$I(U;\Pi )\le I(U;\Pi ,\mathbf{X})\le \frac{1}{2}D(P_{\mathbf{X}\Pi }^{+{\rho }_0}\Vert P_{\mathbf{X}\Pi }^0)+\frac{1}{2}D(P_{\mathbf{X}\Pi }^{-{\rho }_0}\Vert P_{\mathbf{X}\Pi }^0),$$

$$D(P\Vert Q)=\int \log \left(\frac{dP}{dQ}\right)dP$$

$$I(X;Y)=D(P_{XY}\Vert P_X\times P_Y),$$

$$I(X;Y)=\underset{Q_Y}{\inf }D(P_{Y|X}\Vert Q_Y|P_X),$$

$$I(X;Y|Z)=D(P_{XY|Z}\Vert P_{X|Z}\times P_{Y|Z}|P_Z)$$

$$I(U_1;Y)+I(U_2;X|U_1)+I(U_3;Y|U^2)+\dots$$

$$\le s_{\infty }[I(U_1;X)+I(U_2;Y|U_1)+I(U_3;X|U^2)+\dots ]$$

$$U_r-(X,U^{r-1})-Y,$$

$$U_r-(Y,U^{r-1})-X,$$

$$I(X;Y)-I(X;Y|U)\le s_{\infty }I(U;X,Y).$$

$$P_{XY|U^r=u^r}(x,y)=f(x)P_{XY}(x,y)g(y),$$

$$P_{XY}(x,y)=f(x)g(y)Q_{XY}(x,y).$$

$$I(U;Y)$$

$$I(X;V)$$

$${\mathrm{I}}_F(X;\theta )\triangleq \int {\left(\frac{\partial }{\partial \theta }\ln \frac{dP(\cdot |\theta )}{d\mu }(x)\right)}^{2}dP(x|\theta ).$$

$$g(\theta ,ϵ)\triangleq D\left(P(x|\theta )\Vert P(x|\theta +ϵ)\right)$$

$${\mathrm{I}}_F(X;\theta )=\ln 2\cdot {\frac{{\partial }^{2}g(\theta ,\epsilon )}{\partial {\epsilon }^2}|}_{\epsilon =0},$$

$${\mathrm{I}}_F(X;\theta )=2\ln 2\cdot \underset{\epsilon \to 0}{\lim }\frac{g(\theta ,\epsilon )}{{\epsilon }^2}.$$

$$\frac{d{P}^{\rho }}{d\mu }(\Pi ,\mathbf{x})=\frac{{\sum }_{\mathbf{y}}P(\Pi |\mathbf{x},\mathbf{y})P^{\rho }(\mathbf{x},\mathbf{y})}{{\sum }_{\mathbf{y}}P(\Pi |\mathbf{x},\mathbf{y})P^0(\mathbf{x},\mathbf{y})}\le \underset{\mathbf{x},\mathbf{y}}{\sup }\frac{P^{\rho }(\mathbf{x},\mathbf{y})}{P^0(\mathbf{x},\mathbf{y})}$$

$$\frac{d{P}^{\rho }}{d\mu }(\Pi ,\mathbf{x})$$

$$\frac{{\partial }^m}{\partial {\rho }^m}\left[\frac{d{P}^{\rho }}{d\mu }(\Pi ,\mathbf{x})\right]$$

Full text

1 Introduction

The problem of distributed statistical inference under communication constraints has gained much recent interest in the theoretical computer science, statistics, machine learning, and information theory communities. The prototypical setup involves two or more remote parties, each observing local samples drawn from of a partially known joint statistical model. The parties are interested in estimating some well-defined statistical property of the model from their data, and to that end, can exchange messages under some prescribed communication model. The communication complexity associated with this estimation problem concerns the minimal number of bits that need to be exchanged in order to achieve a certain level of estimation accuracy. Whereas the sample-complexity of various estimation problems in the centralized case is well studied (see e.g. [LC06],[VT04]), the fundamental limits of estimation in a distributed setup are far less understood, due to the inherent difficulty imposed by the restrictions on the communication protocol.

In this paper, we study the following distributed estimation problem. Alice and Bob observe infinitely many iid copies of $\rho$-correlated unit variance random variables, that are either binary symmetric or Gaussian, and where the correlation $\rho\in[-1,1]$ is unknown. By interactively exchanging $k$ bits on a shared blackboard, Bob wants to produce an estimate $\hat{\rho}$ that is guaranteed to be $\epsilon$-close to $\rho$ (in the sense that $\mathbb{E}[(\hat{\rho}-\rho)^{2}]\leq\epsilon^{2}$) regardless of the true underlying value of the correlation. We show that the communication complexity of this task, i.e., the minimal number of bits $k$ that need to be exchanged between Alice and Bob to that end, is $\frac{1+o(1)}{2\epsilon^{2}\ln 2}$ in both the binary and Gaussian settings, and one-way schemes are optimal. We also prove a local version of the bound, showing that the communication complexity is still $\Theta_{\rho}(1/\epsilon^{2})$ even if the real correlation is within an interval of vanishing size near $\rho$.

Let us put our work in context of other results in the literature. The classical problem of communication complexity, originally introduced in a seminal paper by Yao for two parties [Yao79], has been extensively studied in various forms and variations, see e.g. [KN96] and references therein. In its simplest (two-party) form, Alice and Bob wish to compute some given function of their local inputs, either exactly for any input or with high probability over some distribution on the inputs, while interactively exchanging the least possible number of bits. While in this paper we also care about the communication complexity of the task at hand, our setting differs from the classical setup in important ways. First, rather than computing a specific function of finite input sequences with a small error probability under a known distribution (or in the worst case), we assume an unknown parametric distribution on infinite input sequences, and wish to approximate the underlying parameter to within a given precision. In a sense, rather than to compute a function, our task is to interactively extract the most valuable bits from the infinite inputs towards our goal. Notwithstanding the above, an appealing way to cast our problem is to require interactively approximating the function

[TABLE]

for infinite iid (binary or Gaussian) strings $\mathbf{X},\mathbf{Y}$, to within precision $\epsilon$, which we show requires $\Theta(1/\epsilon^{2})$ bits of interactive communication.

In another related aspect, many traditional communication complexity lower bounds are proved via information-theoretic arguments, most notably by bounding the information complexity of good protocols over a suitable choice of distribution over the inputs, see e.g. the classical proof for the disjointness problem [BYJKS04]. Our proofs have a similar information-theoretic flavor; in fact, our key technical contribution is connecting a so-called symmetric strong data-processing inequality (SDPI), previously considered in [LCV17] in the context of interactive secret key generation, to interactive hypothesis testing and estimation problems. Loosely speaking, the symmetric SDPI gives the following bound:

[TABLE]

This is formalized in our Lemmas 5, 7 and 8, where the upper and lower expressions above correspond to $R$ in eq. (88) and $S$ in eq. (89), respectively. In fact, as a side application of this interactive SDPI, we show an $\Omega(n)$ lower bound on information complexity of the Gap-Hamming problem [IW03],[CR12], which has so far resisted an information-theoretic attack; see Remark 3 for details.

There has also been much contemporary interest in distributed estimation with communication constraints under a different context, where a finite number of iid samples from a distribution belonging to some parametric family are observed by multiple remotely located parties, which in turn can communicate with a data center (either one-way or interactively) in order to obtain an estimate of the underlying parameters, under a communication budget constraint, see e.g. [ZDJW13], [BGM*+*16], [HÖW18]. These works are markedly different from ours: the samples observed by the parties are taken from the same distribution, and the main regime of interest is typically where the dimension of the problem is relatively high compared to the number of local samples (so that each party alone is not too useful), but is low relative to the total number of samples observed by all parties (so that centralized performance is good). The goal is then to communicate efficiently in order to approach the centralized performance. This stands in contrast to our case, where each party observes an unlimited number of samples drawn from a different component of a bivariate, single-parameter distribution, and the difficulty hence lies in the fact that the quantity of interest (correlation) is truly distributed; none of the parties can estimate it alone, and both parties together can estimate it to arbitrary precision in a centralized setup. Hence, the only bottleneck is imposed by communications.

Another line of works closely related to ours has appeared in the information theory literature, limited almost exclusively to one-way protocols. The problem of distributed parameter estimation under communication constraints has been originally introduced in [ZB88], where the authors provided a rate-distortion-type upper bound on the quadratic error in distributively estimating a scalar parameter using one-way communication (possibly to a third party) under a rate constraint in communication-bits per sample, for a limited set of estimation problems. They have studied our Gaussian setup in particular, and the upper bounds we briefly discuss herein can be deduced (albeit non-constructively) from their work (in [HS18] it is shown how to constructively attain the same performance, and also generalize to the vector parameter case). There has been much followup work on this type of problems, especially in the discrete samples case, see [HA98] and references therein. A local and global minimax setup similar to our own (but again for one-way protocols) has been explored in [AB90]. The local minimax bound we obtain (for one-way protocols) was essentially claimed in that paper, but a subtle yet crucial flaw in their proof of the Fisher information tensorization has been pointed out later in [HA98].

Finally, it is worth noting the closely related problem of distributed hypothesis testing for independence under communication constraints. In [AC86], the authors provide an exact asymptotic characterization of the optimal tradeoff between the rate (in bits per sample) of one-way protocols and the false-alarm error exponent attained under a vanishing mis-detect probability. This result has recently been extended to the interactive setup with a finite number of rounds [XK13].

In fact, some of our lower bounds are also based on a reduction to testing independence with finite communication complexity. For a special case of one-way protocols, this problem was recently analyzed in [ST18]. There is also an inherent connection between the problem of testing independence and generating common randomness from correlated sources, cf. [TW15], as well as between the problem of testing independence and hypercontractivity [Pol12]. For common randomness, two recent (and independent) works [GR16, LCV17] showed that to (almost) agree on $L$ (almost) pure bits the minimal two-way communication required is $(1-\rho^{2})L$. There are several differences between the results and techniques in these two works. The work [GR16] followed upon earlier [CGMS17] and considers exponentially small probability of error. Their main tool is hypercontractivity (Concurrently, hypercontractivity bounds for one-way protocols in similar common randomness generation models were also obtained independently in [LCV15][LCCV16]). The lower bound in [GR16] was partial, in the sense that the common randomness generated by Alice was required to be a function of her input and not of the transcript. Thus [GR16] [LCV15][LCCV16] all concern settings where one-way protocols are optimal. In contrast, the work [LCV17] followed on a classical work on interactive compression [Kas85] and showed an unrestricted lower bound. In that setting, one-way communication is not optimal for general sources (although it was numerically verified and proved in the limiting cases that one-way protocols are optimal for binary symmetric sources). The main tool in [LCV17] in the small communication regime was the “symmetric strong data-processing inequality”. Here we adapt the latter to our problem.

Organization. In Section 2 we formally present the problem and state our main results. Section 3 contains necessary mathematical background. Section 4 proves that the achievability in the Gaussian case implies the achievability in the binary symmetric case (so that we only need to the achievability for Gaussian and converse for binary). Section 5 proves the upper bounds. Section 6 proves the lower bounds in the special case of one-way protocols (as a warm-up), and Section 7 proves the lower bound in the full interactive case, both for the global risks. Section 8 discusses how to extend to the local version by using common randomness. Section 9 gives the technical proof for the symmetric strong data processing inequality in the binary and Gaussian cases.

2 Main results

We define the problem formally as follows. Alice and Bob observe $\mathbf{X}$ and $\mathbf{Y}$ respectively, where $(\mathbf{X},\mathbf{Y})\sim P_{XY}^{\otimes n}$. The distribution $P_{XY}$ belongs to one of the two families, parameterized by a single parameter $\rho\in[-1,1]$:

Binary case: $X,Y\in\{\pm 1\}$ are unbiased and $\mathbb{P}[X=Y]=\tfrac{1+\rho}{2}$. 2. 2.

Gaussian case: $X,Y$ are unit-variance $\rho$-correlated Gaussian.

The communication between Alice and Bob proceeds in rounds: First, Alice writes $W_{1}=f_{1}(\mathbf{X})$ on the board. Bob then writes $W_{2}=f_{2}(\mathbf{Y},W_{1})$ and so on where in the $r$-th round Alice writes $W_{r}$ if $r$ is odd, and Bob writes $W_{r}$ if $r$ is even, where in both cases $W_{r}=f_{r}(\mathbf{X},W_{1},\ldots,W_{r-1})$. We note that, in principle, we allow each function $f_{r}$ to also depend on a private randomness (i.e. $f_{r}$ can be a stochastic map of its arguments). We also note that our impossibility results apply to a slightly more general model where there is also a common randomness in the form of a uniform $W_{0}$ on $[0,1]$ pre-written on the board, but we do not need this for our algorithms.

Let $\Pi=(W_{1},W_{2},\ldots)$ be the contents of the board after all of (possibly infinitely many) rounds. We say that the protocol is $k$-bit if the entropy $H(\Pi)\leq k$ for any $\rho\in[-1,1]$. Note that the protocol is completely characterized by the conditional distribution $P_{\Pi|\mathbf{X}\mathbf{Y}}$.

At the end of communication, Bob produces an estimate $\hat{\rho}(\Pi,\mathbf{Y})$ for the correlation $\rho$ of the underlying distribution. We are interested in characterizing the tradeoff between the communication size $k$ and the worst-case (over $\rho$) squared-error, which we call quadratic risk, in the regime where the number of samples $n$ is arbitrarily large but $k$ is fixed. Explicitly, the quadratic risk of the protocol $\Pi$ and the estimator $\hat{\rho}$ is given by

[TABLE]

where $\operatorname{\mathbb{E}}_{\rho}$ is the expectation under the correlation value $\rho$. Similarly, we write $P^{\rho}_{\mathbf{X}\mathbf{Y}\Pi}$ for the joint distribution corresponding to a fixed value of $\rho$. The (global) minimax risk is defined as

[TABLE]

whereas the local minimax risk is

[TABLE]

The infima in both the definitions above are taken over all $k$-bit protocols $\Pi$ and estimators $\hat{\rho}$, as well as the number of samples $n$. We will also discuss one-way protocols, i.e. where $\Pi=W_{1}$ consists of a single message from Alice to Bob. We denote the global and local minimax risk in the one-way case by $R^{*1}$ and $R^{*1}_{\rho,\delta}$ respectively.

Our main results are the following.

Theorem 1** (Upper bounds).**

In both the Gaussian and the binary symmetric cases with infinitely many samples,

[TABLE]

as long as $\delta=o(1)$ (here and after, $o(1)$ means a vanishing sequence indexed by $k$), and

[TABLE]

In fact, one-way protocols achieve these upper bounds.

Remark 1**.**

Previously, [HS18] showed that there exists a one-way protocol and an unbiased estimator achieving $R_{\rho}(\Pi,\hat{\rho})\leq\frac{1}{k}\left(\frac{1-\rho^{2}}{2\ln 2}+o(1)\right)$ for any $\rho$. The protocol (in the Gaussian case) sends the index $\operatorname*{argmax}_{1\leq i\leq 2^{k}}X_{i}$ using $k$ bits and employs the super concentration property of the max. Here, the local risk bound (5) is tighter because we can send the index more efficiently using the side information $Y^{2^{k}}$ and the knowledge of $\rho$ within $o(1)$ error. Such a scheme has the drawback that it is specially designed for a small interval of $\rho$ (as in the definition of the local risk), and hence the performance may be poor outside that small interval. However, we remark that one can achieve the risk $\frac{1}{k}\left(\frac{\left(1-\rho^{2}\,\right)^{2}}{2\ln 2}+o(1)\right)$ at any $\rho$ by a two-way protocol. Indeed, Alice can use the first round to send $\omega(1)\cap o(k)$ bits to Bob so that Bob can estimate $\rho$ up to $o(1)$ error. Then Bob can employ the one-way local protocol in (5) for the $\rho$ estimated from the first round.

Theorem 2** (lower bounds).**

In both the Gaussian and binary symmetric cases with infinitely many samples,

[TABLE]

In particular, since the global risk dominates the local risk at any $\rho$, we have

[TABLE]

Note in particular that theorems Theorem 1 and 2 have identified the exact prefactor in the global risk.

Remark 2** (Unbiased estimation).**

We note that the proof of Theorem 2 also implies that for any unbiased estimator $\hat{\rho}$ of $\rho$ in the binary case it holds that

[TABLE]

We further note that an unbiased estimator with $\operatorname{\mathsf{Var}}\hat{\rho}=(1-\rho^{2}+o(1))/(2k\ln 2)$ was introduced in [HS18] (and discussed in Section 5 below), establishing the tightness of (9) at $\rho=0$.

The bound (9) follows from the Cramér-Rao inequality (see e.g. [VT04]) along with the bound we obtain for the Fisher information given in (73). The associated regularity conditions are discussed in Remark 4.

Remark 3** (Gap-Hamming Problem).**

In the Gap-Hamming problem [IW03], Alice and Bob are given binary length-$n$ vectors $(\mathbf{X}$ and $\mathbf{Y})$ respectively, with the promise that $\#\{i:X_{i}\neq Y_{i}\}$ is either $\leq n/2-\sqrt{n}$ or $\geq n/2+\sqrt{n}$. They communicate in (possibly infinitely many) rounds to distinguish these two hypotheses. It was shown in [CR12] (later with simplified proofs in [Vid12], [She12]) that the communication complexity of any protocol that solves Gap-Hamming with small error probability is $\Omega(n)$ (an upper bound of $O(n)$ is trivial). However, whereas many interesting functions in communication complexity have information-theoretic lower bounds, Gap-Hamming has so far resisted an information-theoretic proof, with the exception of the single-round case for which a proof based on SDPI is known [Gar18]. It is nevertheless already known that the information complexity of Gap-Hamming is linear, i.e., that $I(\Pi;\mathbf{X},\mathbf{Y})=\Omega(n)$ for any $\Pi$ that solves it, under the uniform distribution on $(\mathbf{X},\mathbf{Y})$. This is however observed only indirectly, since the smooth rectangle bound used in the original proof is known to be “below” the information complexity, i.e., any lower bound proved using the smooth rectangle bound also yields a lower bound on information complexity. It is therefore of interest to note that our result in particular directly implies a $\Omega(n)$ lower bound on the information complexity of Gap-Hamming (and hence also on its communication complexity).

To see this, we note that the main step in proving our main result is the inequality

[TABLE]

which is implied by Theorems 4 and 5, in Section 7. We note that it implies the $\Omega(n)$ lower-bound on distributional communication and information complexity of the Gap-Hamming problem, see [CR12] for references and the original proof of $\Omega(n)$. Indeed, let $U\sim\mathrm{Ber}(1/2)$ and given $U$ let $\mathbf{X},\mathbf{Y}$ have correlation $\rho=(-1)^{U}\rho_{0}$, where $\rho_{0}=\tfrac{100}{\sqrt{n}}$. Take $\Pi$ to be a protocol used for solving the Gap-Hamming problem (which decides whether $\#\{i:X_{i}\neq Y_{i}\}$ is $\leq n/2-\sqrt{n}$ or $\geq n/2+\sqrt{n}$ with small error probability). Its decision should equal $U$ with high probability, and hence there exists a decision rule based on $\Pi$ reconstructing $U$ with high probability of success. Thus $I(U;\Pi)=\Omega(1)$, and we further have

[TABLE]

where the last inequality follows from a property of the mutual information ((14) below). Finally, from (10) we get the statement that $H(\Pi)\geq\Omega(\rho_{0}^{-2})=\Omega(n)$.

As pointed out by the anonymous reviewer, a more general version of the Gap-Hamming problem concerns the decision between $\#\{i\colon X_{i}\neq Y_{i}\}\leq n/2-g$ and $n/2+g$ for some $\sqrt{n}\leq g\leq n/2$, and it was shown in [CR12, Proposition 4.4] that the communication complexity is $\Omega(n^{2}/g^{2})$. This result can also be recovered by the above argument. And notably, this result also implies the $R^{*}=\Omega(1/k)$ lower bound.

3 Preliminaries

3.1 Notation

Lower- and upper-case letters indicate deterministic and random variables respectively, with boldface used to indicate $n$-dimensional vectors. For any positive integer $r$, the set $\{1,2,\dots,r\}$ is denoted by $[r]$. Let $P$ and $Q$ be two probability distributions over the same probability space. The KL divergence between $P$ and $Q$ is

[TABLE]

with the convention that $D(P\|Q)=\infty$ if $P$ is not absolutely continuous w.r.t. $Q$. Logarithms are taken to the base $2$ throughout, unless otherwise stated. With this definition, the mutual information between two jointly distributed r.v.s $(X,Y)\sim P_{XY}$ can be defined

[TABLE]

and it satisfies the “radius” property:

[TABLE]

where the conditioning means taking the expectation of the conditional KL divergence w.r.t. $P_{X}$. Given a triplet of jointly distributed r.v.s $(X,Y,Z)$, the conditional mutual information between $X$ and $Y$ given $Z$ is

[TABLE]

We will say that r.v.s $A,B,C$ form a Markov chain $A-B-C$, if $A$ is independent of $C$ given $B$.

3.2 Symmetric strong data-processing inequalities

Given $P_{XY}$, the standard data processing inequality states that $I(U;Y)\leq I(U;X)$ for any $U$ satisfying $U-X-Y$. Recall that a strong data processing inequality (see e.g. [PW17]) is satisfied if there exists $s\in[0,1)$ depending on $P_{XY}$ such that $I(U;Y)\leq sI(U;X)$ for any $U$ satisfying $U-X-Y$.

The connection between the strong data processing and communication complexity problems is natural, and $U$ can be thought of as the message from Alice to Bob, $I(U;X)$ the communication complexity, and $I(U;Y)$ the information for the estimator. However, the best constant $s$ in the strong data processing inequality is not symmetric (i.e. $s(P_{XY})=s(P_{YX})$ is not true for general $P_{XY}$), whereas the performance in an interactive communication problems is by definition symmetric w.r.t. the two parties. An inequality of the following form, termed “symmetric strong data processing inequality” in [LCV17], plays a central role in interactive communication problems:

[TABLE]

where $U_{1}$, $U_{2}$, …must satisfy

[TABLE]

and where $s_{\infty}$ depends only on $P_{XY}$. Clearly $s_{\infty}(P_{XY})=s_{\infty}(P_{YX})$ and $s_{\infty}\geq s$. A succinct characterization of $s_{\infty}$ in terms of the “marginally convex envelope” was reported in [LCV17]. Using the Markov assumptions (17)-(18) we can also rewrite (16) as

[TABLE]

When $X,Y$ are iid binary symmetric vectors with correlation $\rho^{2}$ per coordinate, it was shown in [LCV17] that $s_{\infty}=\rho^{2}$, equal to the strong data processing constant. In this paper, we extend the result to Gaussian vectors with correlation $\rho$ per coordinate (Theorem 5).

In order to upper bound $s_{\infty}$ in the binary case, [LCV17] observed that $s_{\infty}(Q_{XY})$ is upper bounded by the supremum of $s(P_{XY})$ over $P_{XY}$ a “marginally titled” version of $Q_{XY}$. Indeed, note that the Markov structure in the strong data processing inequality implies that $P_{XY|U=u}(x,y)=f(x)P_{XY}(x,y)$ for some function $f$. In the case of symmetric strong data processing inequality, the Markov conditions (17) and (18) imply that

[TABLE]

which naturally lead one to considering the following result:

Lemma 1** ([LCV17, Theorem 6]).**

Let $Q_{XY}$ be the distribution of a binary symmetric random variables with correlation $\rho\in[-1,1]$, i.e. $Q_{XY}(x,y)=\tfrac{1}{4}(1+(-1)^{1\{x\neq y\}}\rho)$ for $x,y\in\{0,1\}$. Let $(X,Y)\sim P_{XY}$ have an arbitrary distribution of the form

[TABLE]

Then for any $U-X-Y-V$ we have

[TABLE]

Lemma 1 was proved in [LCV17] by exploring the connection to the maximal correlation coefficient. In Section 9.1 we give another proof using properties of the strong data processing inequalities [PW17].

3.3 Fisher information and Cramèr-Rao inequalities

We recall some standard results from parameter estimation theory. Let $\theta$ be a real-valued parameter taking an unknown value in some interval $[a,b]$. We observe some random variable (or vector) $X$ with distribution $P(x|\theta)$ parameterized by $\theta$.

Assume that $P(\cdot|\theta)$ is absolutely continuous with respect to a reference measure $\mu$, for each $\theta\in[a,b]$, and $\frac{dP(\cdot|\theta)}{d\mu}(x)$ is differentiable with respect to $\theta\in(a,b)$ for $\mu$-almost all $x$. Then the Fisher information of $\theta$ w.r.t. $X$, denoted as $\mathrm{I}_{F}(X;\theta)$, is the variance of the derivative of the log-likelihood w.r.t. $\theta$,

[TABLE]

We now record some useful facts concerning the Fisher information. First, we recall that the Fisher information encodes the curvature of the KL divergence w.r.t. translation: Let

[TABLE]

for any $\theta,\theta+\varepsilon\in(a,b)$. The following property is well-known:

Lemma 2**.**

Under suitable regularity conditions, $\frac{\partial}{\partial\varepsilon}g(\theta,\varepsilon)|_{\epsilon=0}=0$, and

[TABLE]

which implies that

[TABLE]

Remark 4**.**

The “regularity conditions” in Lemma 2 (and Lemma 3 below) are to ensure that one can apply the dominated convergence theorem to exchange certain integrals and differentiations in the calculus. See for example [Kul, Section 2.6] for details. In particular, these conditions are fulfilled if $\sup_{x,\theta}\frac{dP(\cdot|\theta)}{d\mu}(x)<\infty$, $\inf_{x,\theta}\frac{dP(\cdot|\theta)}{d\mu}(x)>0$, and $\sup_{x,\theta}\frac{\partial^{m}}{\partial\theta^{m}}\left[\frac{dP(\cdot|\theta)}{d\mu}(x)\right]<\infty$ for $m=1,2,3$. In the interactive estimation problem, these conditions are always satisfied for sources $(\bf X,Y)$ on finite alphabets (even if the message alphabets are not finite). Indeed, suppose that $\bf X,Y$ are binary vectors, and that Alice performs an estimation. Let the reference measure $\mu=P^{0}(\Pi,\bf X)$ be the distribution under $\rho=0$. We have that

[TABLE]

is bounded by a value independent of $\Pi$. Similarly,

[TABLE]

The Fisher information can be used to lower bound the expected quadratic risk of estimating $\theta$ from $X$ under a prior distribution on $\theta$.

Lemma 3** (Bayesian Cramér-Rao inequality, see e.g. [VT04]).**

Let $\lambda$ be an absolutely continuous density on a closed interval $\mathcal{J}\subseteq[a,b]$, and assume $\lambda$ vanishes at both endpoints of $\mathcal{J}$. If $P(x|\theta)$ satisfies suitable regularity conditions and $\mathrm{I}_{F}(X;\theta)<\infty$ for almost all $\theta$,

[TABLE]

for any estimator $\hat{\theta}$, where $\mathrm{I}^{\lambda}=\int_{\mathcal{J}}\frac{\lambda^{\prime 2}}{\lambda}d\theta$.

A common choice of prior (see e.g. [Tsy09]) is

[TABLE]

where $\theta_{0}$ is the center of the interval $\mathcal{J}$, and $\lambda_{0}(x)=\cos^{2}(\pi x/2)$ for $-1\leq x\leq 1$ and [math] otherwise. This prior satisfies $\mathrm{I}^{\lambda}=(2\pi/|\mathcal{J}|)^{2}$.

4 Reduction of binary to Gaussian

In this section we show that an achievability scheme for iid Gaussian vector can be converted to a scheme for binary vector by a preprocessing step and applying the central limit theorem (CLT). We remark that a similar argument was used in [LCV17] in the context of common randomness generation.

Lemma 4**.**

Suppose that $(\Pi,\hat{\rho})$ is a scheme for iid sequence of Gaussian pairs at some length $n$, and the message alphabet size $|\Pi|<\infty$. Then there exists a scheme $(\Pi^{T},\hat{\rho}^{T})$ for iid sequence of binary symmetric pairs of length $T$, for each $T=1,2,\dots$, such that

[TABLE]

where $\rho$ denotes the correlation of the Gaussian or binary pair.

Proof.

Let $(A_{l},B_{l})_{l=1}^{t}$ be an iid sequence of binary symmetric random variables with correlation $\rho$, and put

[TABLE]

where $N$ and $N^{\prime}$ are standard Gaussian random variables, and $N$, $N^{\prime}$, $(X^{t},Y^{t})$ are independent. By the central limit theorem, we can choose some $a_{t}=o(1)$ such that the distribution of $(X^{(t)},Y^{(t)})$ converges to the Gaussian distribution $P_{XY}$ in total variation (Proposition 1 below). Now let $T=nt$ and suppose that $(A_{l},B_{l})_{l=1}^{T}$ is an iid sequence of binary symmetric pairs. The above argument shows that Alice and Bob can process locally to obtain iid sequence of length $n$, which convergences to the iid sequence of Gaussian pairs of correlation $\rho$ in the total variation distance. After this preprocessing step, Alice and Bob can apply the given scheme $(\Pi,\hat{\rho})$. Then (32) follows since entropy is continuous w.r.t. the total variation on finite alphabets, and (33) follows since we can assume without loss of generality that $\hat{\rho}$ is bounded. Note that we have constructed $(\Pi^{T},\hat{\rho}^{T})$ only for $T$ equal to a multiple of $n$; however this restriction is obviously inconsequential. ∎

Proposition 1**.**

There exist $a_{t}=o(1)$ such that $X^{(t)}$ and $Y^{(t)}$ defined in (34) and (35) converges to the Gaussian distribution $P_{XY}$ in total variation.

Proof.

By the convexity of the relative entropy, we can upper bound the KL divergence by the Wasserstein 2 distance:

[TABLE]

However, $\frac{A_{1}+\dots+A_{t}}{\sqrt{t}}$ and $\frac{B_{1}+\dots+B_{t}}{\sqrt{t}}$ converge to $P_{XY}$ under Wasserstein 2 distance, since this is equivalent to convergence in distribution in the current context where a uniformly integrable condition is satisfied (see e.g. [Vil03, Theorem 7.12]) 111Alternatively, see [MT74] for a direct proof of the central limit theorem under the Wasserstein metric.. Thus there exists $a_{t}=o(1)$ such that (36) vanishes. By Pinsker’s inequality, this implies that $(X^{(t)},Y^{(t)})$ converges to the Gaussian distribution $(X+a_{t}N,Y+a_{t}N^{\prime})$ in total variation. However, as long as $a_{t}=o(1)$ we have that $(X+a_{t}N,Y+a_{t}N^{\prime})$ converges to $(X,Y)$. The conclusion then follows by the triangle inequality of the total variation. ∎

5 Proof of the upper bounds (Theorem 1)

Before the proof, let us observe the suboptimality of a naive scheme. Consider the binary case for example (the Gaussian case is similar). Suppose that Alice just sends her first $k$ samples $X_{1},\ldots,X_{k}$. This would let Bob, by computing the empirical average $\hat{\rho}_{\textnormal{emp}}=\tfrac{1}{k}\sum_{j}X_{j}Y_{j}$, achieve a risk of

[TABLE]

Clearly (37) is not sufficient for the upper bounds in Theorem 1. To improve it, we now recall the “max of Gaussian scheme” in [HS18]. By a central limit theorem argument we can show that binary estimation is easier than the Gaussian counterpart (see Lemma 4). Hence we only need to prove the achievability for the Gaussian case. Alice observes the first $2^{k}$ Gaussian samples, and transmits to Bob, using exactly $k$ bits, the index $W$ of the maximal one, i.e.

[TABLE]

Upon receiving the index $W$, Bob finds his corresponding sample $Y_{W}$ and estimates the correlation using

[TABLE]

Recall the following result [HS18], for which we reproduce the short proof and then explain how the local upper bound will follow with a modification of the proof.

Theorem 3** ([HS18]).**

The estimator $\hat{\rho}_{\textnormal{max}}$ is unbiased with

[TABLE]

Proof.

It is easy to check that $\hat{\rho}_{\textnormal{max}}$ is unbiased. In order to compute its variance, we need to compute the mean and variance of $X_{W}$, which is the maximum of $2^{k}$ iid standard normal r.v.s. From extreme value theory (see e.g. [DN04]) applied to the normal distribution, we obtain

[TABLE]

Therefore, for $Z\sim\mathcal{N}(0,1)$ we have that

[TABLE]

∎

Taking $\rho=0$ in (40) establishes the global upper bound (6). However, achieving the local risk upper bound in (5) is trickier, since a direct application of (40) is loose by a factor of $(1-\rho^{2})$. The trick is to send the index $W$ more efficiently using the side information. More precisely, Alice looks for the maximum sample out of $2^{k}$ samples as before. Bob sifts his corresponding samples, marking only those where $Y_{k}>\rho\cdot\sqrt{k\cdot 2\ln 2}\cdot(1-o(1))$. Note that here $\rho$ is as in the definition of the local risk (4), and the true correlation is within $o(1)$ error to $\rho$. It is easy to check that with sufficiently high probability (sufficiently here (and below) meaning that the complimentary probability has a negligible effect on the ultimate variance), there are $2^{k(1-\rho^{2})(1+o(1))}$ such marked samples that also include the one corresponding to Alice’s maximum. Also, by symmetry these marked samples are uniformly distributed among the total $2^{k}$ samples. Hence, Alice can describe the $k\cdot(1-\rho^{2})\cdot(1+o(1))$ most significant bits (say) of the index of her maximal sample, which will reveal this index to Bob with sufficiently high probability. This yields a $(1-\rho^{2})$ factor saving in communication, and the claim follows.

Remark 5**.**

We note that the above risk can also be achieved directly in Hamming space (without appealing to the CLT). Alice sets some parameter $\tilde{\rho}\in[-1,1]$ to be optimized later, and partitions her data to $m$ blocks of size $n$. She then finds the first block whose sum is exactly $n\tilde{\rho}$ (recall the samples are in $\{-1,1\}$), which exists with sufficiently high probability for $m=2^{n(\frac{1}{2}-h(\frac{1-\tilde{\rho}}{2})+o(1))}$ (otherwise, she picks the first block). Bob sifts his corresponding blocks, marking only those with sum $n\rho\tilde{\rho}(1+o(1))$. Alice encodes the index of her chosen block using $\log m=n(\frac{1}{2}-h(\frac{1-\tilde{\rho}}{2})+o(1))$ bits, and sends only the $n(h(\frac{1-\rho\tilde{\rho}}{2})-h(\frac{1-\tilde{\rho}}{2}))$ most significant bits, so that Bob can resolve the index with sufficiently high probability. Bob then finds the sum of his corresponding block, and divides it by $n\tilde{\rho}$ to obtain his estimator for $\rho$. It is straightforward to check that this procedure results in a variance of $\frac{1}{k}\cdot\left((1-\rho^{2})(h(\frac{1-\rho\tilde{\rho}}{2})-h(\frac{1-\tilde{\rho}}{2}))/\tilde{\rho}^{2}+o(1)\right)$, where $h(q)=-q\log_{2}{q}-(1-q)\log_{2}(1-q)$ is the binary entropy function. Optimizing over $\tilde{\rho}$ yields that $\tilde{\rho}\to 0$ is (not surprisingly) optimal, and the obtained variance is the same as the one achieved by the modified Gaussian maximum estimator above.

6 Proof of the lower bounds in Theorem 2 (one-way case)

In this section we prove the lower bounds on the global and local one-way risks, $R^{*1}$ and $R^{*1}_{\rho,\delta}$, in the binary case. The Gaussian case will then follow from the central limit theorem argument in Lemma 4. Of course, the one-way lower bound is a special case of the interactive case in Section 7; we separate the discussion simply because the one-way case is conceptually easily and the proof does not need the symmetric strong data processing inequality (modulus certain technical issues pertaining the continuity of Fisher information which we will discuss).

We note that in the one-way setting, the following Markov chain holds:

[TABLE]

Note that regardless of $\rho$ the marginal distribution of $\mathbf{X}$ (and thus of $\Pi$) is the same. Let $P_{\Pi\mathbf{Y}}^{\rho}$ denote the joint distribution of $(\Pi,\mathbf{Y})$ when the correlation is equal to $\rho$. Note that under $\rho=0$ we have that $\Pi$ and $\mathbf{Y}$ are independent. Thus, via (13) we obtain

[TABLE]

Furthermore, from (91) we get

[TABLE]

Thus using the connection between the KL divergence and the Fisher information in Lemma 2, we obtain

[TABLE]

Now, suppose that we can show a continuity result for the Fisher information at $\rho=0$, in the sense of

[TABLE]

then a standard application of the Bayesian Cramér-Rao bound would imply the global risk. Indeed, applying Lemma 3 with (e.g.) the prior specified in (31) over $\mathcal{J}=[\rho-\delta,\rho+\delta]$, we obtain

[TABLE]

for $\delta\in o(1)\cap\omega(1/\sqrt{k})$, establishing (8) for the special case of one-way protocols.

While the continuity claim in (51) is intuitive enough, to rigorously show it we need to resort to a device to be discussed in Section 8, which will allow us to reduce the problem of testing against an arbitrary $\rho$ to testing against independence. Specifically, in Corollary 1 we will show that using common randomness this can be generalized to yield

[TABLE]

for any $\rho_{1}\in[-1,1]$ and $\rho_{0}\in[\tfrac{\rho_{1}-1}{2},\tfrac{\rho_{1}+1}{2}]$. Again applying Lemma 2, we obtain

[TABLE]

for any $\rho\in(-1,1)$. This justifies the continuity claim (51). Moreover, applying the Bayesian Cramér-Rao (Lemma 3) with (e.g.) the prior specified in (31) over $\mathcal{J}=[\rho-\delta,\rho+\delta]$, we obtain

[TABLE]

which is the desired local risk lower bound for the special case of one-way protocols.

7 Proof of lower bounds in Theorem 2 (interactive case)

For the interactive case, our approach is again to upper bound the KL divergence between the distributions of the r.v.s available (to either Alice or Bob) under $\rho\neq 0$, and under $\rho=0$. This is accomplished by Theorem 4 and Theorem 5 below, which can be viewed as generalizations of (48) and (49).

Theorem 4**.**

Consider an arbitrary interactive protocol $P_{\Pi|\mathbf{X}\mathbf{Y}}$ and let $P_{\mathbf{X}\mathbf{Y}\Pi}$ be the induced joint distribution. Let $\bar{P}_{\mathbf{X}\mathbf{Y}\Pi}=P_{\mathbf{X}}\times P_{\mathbf{Y}}\times P_{\Pi|\mathbf{X}\mathbf{Y}}$ be the joint distribution induced by the same protocol, but when the $\mathbf{X}$ and $\mathbf{Y}$ are taken to be independent (but with same marginals). Then

[TABLE]

where information quantities are computed with respect to $P_{\mathbf{X}\mathbf{Y}\Pi}$. Moreover, the bound (56) continues to hold also when the protocol $\Pi$ contains an arbitrary common randomness (i.e. public coin) $W_{0}$ independent of $(\mathbf{X},\mathbf{Y})$.

By saying that the protocol contains common randomness we mean that $\Pi=(W_{0},W_{1},\dots,W_{r})$ where $W_{0}$ is the common randomness and $W_{1},\dots,W_{r}$ are the exchanged messages. The extension to the case of common randomness will be useful in Section 8 where we reduce the problem of testing against an arbitrary $\rho$ to testing against independence.

Proof of Theorem 4.

First, since

[TABLE]

it suffices to prove the same upper bound for

[TABLE]

In other words, it suffices to prove the theorem for the case where the common randomness $W_{0}$ is empty. Under this assumption, note that the RHS of (56) is equal to

[TABLE]

where all expectations are taken with respect to $P_{\mathbf{X}\mathbf{Y}\Pi}$ and the last step is by non-negativity of divergence $D(P_{\mathbf{Y}|\Pi=\pi}\|\bar{P}_{\mathbf{Y}|\Pi=\pi,\mathbf{X}=\mathbf{x}})$ for all $\pi,\mathbf{x}$, which in turn uses the Markov chain $\mathbf{X}-\Pi-\mathbf{Y}$ under $\bar{P}_{\mathbf{X}\Pi\mathbf{Y}}$. In all, (64) proves part of (56). To prove the same bound on $D(P_{\Pi\mathbf{Y}}\|\bar{P}_{\Pi\mathbf{Y}})$ we can argue by symmetry (it may seem that symmetry is broken by the fact that $\mathbf{X}$ sends $W_{1}$ first, but this is not true: $W_{1}$ can be empty), or just perform a straightforward modification of step (62). ∎

Remark 6**.**

It can be seen that for a one-way protocol we have equality in (56). This explains why our impossibility bound can be essentially achieved by a one-way protocol (e.g., see Theorem 3), and suggests that this is the only possibility.

Remark 7**.**

After completion of this work, we found out that in a slightly different form Theorem 4 has previously appeared in [XK13, Equation (4)]. Our proof is slightly simpler.

Theorem 5**.**

Let $\Pi$ be any interactive protocol, possibly containing a common randomness $W_{0}$, in either the Gaussian or the binary symmetric case. Then

[TABLE]

The proof of Theorem 5 is given in Section 9.

Remark 8**.**

The following notions of external and internal information costs were introduced in [CSWY01] and [BBCR10] respectively:

[TABLE]

Using the Markov chain conditions of the messages

[TABLE]

we will be able to write the external and internal information as sums of information gains in each round of communication:

[TABLE]

which are useful later in some proofs.

8 Reduction of testing against arbitrary $\rho$ to testing against independence

The results in Theorem 4 and Theorem 5 only (directly) applies to testing against independence, and hence are insufficient for handling the local risks at an arbitrary $\rho$. Fortunately, for binary and Gaussian vectors, there is a simple device of translating the correlations by leveraging the common randomness, so that the general problem is reduced to the case of testing independence solved in Theorem 4 and Theorem 5. More precisely, we obtained the following result:

Corollary 1**.**

Let $P_{\mathbf{X}\mathbf{Y}}^{\rho_{0}}$ (resp. $P_{\mathbf{X}\mathbf{Y}}^{\rho_{1}}$) be the joint distribution for Gaussian or binary symmetric vector sources under correlation $\rho_{0}$ (resp. $\rho_{1}$). Let $P_{\Pi|\mathbf{X}\mathbf{Y}}$ be an arbitrary protocol. Then for any $\rho_{1}\in[-1,1]$ and $\rho_{0}\in[\tfrac{\rho_{1}-1}{2},\tfrac{\rho_{1}+1}{2}]$,

[TABLE]

In particular, this bounds the Fisher information in the case of finite-length binary vectors as

[TABLE]

Proof.

From Theorems 4 and 5 we have

[TABLE]

The proof uses a device of shifting the correlation by introducing common randomness. Suppose that $\mathbf{X}$ and $\mathbf{Y}$ are iid binary or Gaussian vectors of length $n$, where the correlation between $X_{i}$ and $Y_{i}$ is [math] under $P^{(0)}$ and $\rho:=\frac{\rho_{1}-\rho_{0}}{1-|\rho_{0}|}$ under $P^{(1)}$, for each $i\in[n]$. We define the common randomness $W_{0}$ independent of $\mathbf{X}$ and $\mathbf{Y}$ as follows:

• •

Gaussian case: Let $W_{0}=\mathbf{Z}$, where $Z_{i}\sim\mathcal{N}(0,1)$ are iid, and define

[TABLE]

for some $\alpha\in[-1,1]$ and $s\in\{-1,1\}$.

• •

Binary case: Let $W_{0}=({\bf B,Z})$ where ${\bf B}$ is independent of ${\bf Z}$, $B_{i}\sim\mathrm{Ber}{(\alpha)}$ over $\{0,1\}$ are iid and $Z_{i}\sim\mathrm{Ber}{(\tfrac{1}{2})}$ over $\{-1,1\}$ are iid. Put

[TABLE]

for some $\alpha\in[0,1]$ and $s\in\{-1,1\}$.

In both cases, it can be verified that by appropriately choosing $s$ and $\alpha$, the correlation between $X^{\prime}_{i}$ and $Y^{\prime}_{i}$ equals $\rho_{0}$ under $P^{(0)}$ and $\rho_{1}$ under $P^{(1)}$. Now, consider any protocol $\Pi=(W_{0},W^{r})$ for the source ${\bf X,Y}$ which includes the common randomness $W_{0}$. We have

[TABLE]

where (79) follows since $P^{(1)}_{\mathbf{Y}^{\prime}|W_{0}W^{r}\mathbf{Y}}=P^{(0)}_{\mathbf{Y}^{\prime}|W_{0}W^{r}\mathbf{Y}}=P_{\mathbf{Y}^{\prime}|W_{0}\mathbf{Y}}$ (note $\mathbf{Y}^{\prime}$ is a (deterministic) function of $(W_{0},\mathbf{Y})$), and (80) follows from Theorem 4 and Theorem 5. Observe that $D(P^{(1)}_{W_{0}W^{r}\mathbf{Y}^{\prime}}\|P^{(0)}_{W_{0}W^{r}\mathbf{Y}^{\prime}})$ is exactly $D(P_{\Pi\mathbf{Y}}^{\rho_{1}}\|P_{\Pi\mathbf{Y}}^{\rho_{0}})$ which we wanted to upper bound. Repeating the same steps for $D(P_{\Pi\mathbf{X}}^{\rho_{1}}\|P_{\Pi\mathbf{X}}^{\rho_{0}})$ establishes (72) for both the Gaussian and binary cases.

The bound on the Fisher information in the binary case (73) follows from Lemma 2. ∎

Remark 9**.**

While we expect that the same bound in (73) continues to hold in the Gaussian case, the regularization condition required in the transition from the KL divergence bound to the Fisher information bound appears difficult to justify in the Gaussian case (see Lemma 2 and the ensuing remark).

9 Proof of the symmetric strong data processing inequality

This section proves Theorem 5, which states that the symmetric strong data processing inequality constant is bounded by $\rho^{2}$ in the case of binary symmetric or Gaussian vectors. We first outline the proof, and then supplement the key lemmas used.

Proof of Theorem 5.

First, note that we only need to prove the case where the common randomness $W_{0}$ is empty. Indeed, since $\Pi$ includes $W_{0}$ and since $W_{0}$ is independent of $(\mathbf{X},\mathbf{Y})$, we have $I(\mathbf{X;Y}|\Pi)=I(\mathbf{X;Y}|W_{0},\Pi)$ and $I(\Pi;\mathbf{X,Y})=I(\Pi;\mathbf{X,Y}|W_{0})$, hence (65) will follow if we establish

[TABLE]

for each $w_{0}$. Using the Markov chains satisfied by the messages we have

[TABLE]

Then the result for the binary and Gaussian cases follow respectively from Lemma 5 and Lemma 8, as well as the tensorization property Lemma 7, stated and proved below. ∎

9.1 Binary case

Our goal is to prove Lemma 5, which follows from Lemma 1 stated earlier and proved in this section.

Lemma 5** ([LCV17]).**

Let $X,Y\in\{1,-1\}$ be equiprobably distributed with correlation $\rho$. Consider any random variables $U^{r}$, $r\in\mathbb{Z}$, satisfying

[TABLE]

Define

[TABLE]

Then $R(P_{U^{r}XY})\leq\rho^{2}S(P_{U^{r}XY})$.

Proof.

It suffices to show that the ratio of the $i$-th term on the right side of (88) to the $i$-th term on the right side of (89) is upper-bounded by $\rho^{2}$ for any $i$. Consider without loss of generality any $i\in 2\mathbb{Z}$. Note that by inducting on $i$ and using the Markov chain conditions satisfied by $U^{r}$, we observe that $P_{YX|U^{i-1}=u^{i-1}}$ has the property that

[TABLE]

for some functions $f$ and $g$. Then using Lemma 1 we conclude that for each $u^{i}$ we have $\frac{I(U_{i};X|U^{i-1}=u^{i-1})}{I(U_{i};Y|U^{i-1}=u^{i-1})}\leq\rho^{2}$. ∎

The following result is used in the proof of Lemma 1. We state it in the general vector case, though we only need the scalar $(X,Y)$ case.

Lemma 6**.**

Let $X,Y$ be binary $\mathbb{P}[X=1]=1-\mathbb{P}[X=0]=p$ and let $\mathbb{P}[Y\neq X|X]={1-\rho\over 2}$, $\rho\in[-1,1]$. Consider $(\mathbf{X},\mathbf{Y})$ to be $n$ iid copies of $(X,Y)$. Then for any random variables $U,V$ such that $U-\mathbf{X}-\mathbf{Y}-V$ we have

[TABLE]

Proof.

We first recall that a result known as tensorization (due to [AG76] in this context) allows to only check $n=1$ case. For $n=1$, the first part (91) is the standard inequality dating back to [AG76], see [PW17] for a survey. To show inequality (92), we apply Theorem 21 in [PW17], which establishes the following. Let $A$ be a binary input and $B\sim P$ when $A=0$ and $B\sim Q$ when $A=1$, where $P=(P(v),v=0,1,\ldots)$ and $Q=(Q(v),v=0,1,\ldots)$ are two arbitrary distributions. Then for any $U-A-B$ we have

[TABLE]

(The bound is tight, cf. [PW17, Remark 8], whenever $B$ is binary.) Applying this result to $A=Y$ and $B=X$ and denoting $q=p\rho+{1-\rho\over 2}$ we get

[TABLE]

∎

Proof of Lemma 1.

Due to symmetry, it suffices to prove only the first inequality. Computing $P_{Y|X}$ and applying (93) we need to prove

[TABLE]

where $g_{x}=\sum_{y^{\prime}}g(y^{\prime})Q_{Y|X}(y^{\prime}|x),x\in\{0,1\}$. Note that for all $y$,

[TABLE]

By rescaling $g$ so that $\sum_{y}g(y)=1$ we get that $g_{0}+g_{1}=1$ and hence $\sqrt{g_{0}g_{1}}\leq{1\over 2}$, as required. ∎

9.2 Tensorization

The bound in Lemma 1 does not tensorize. That is, if $Q_{XY}$ in the lemma is replaced by $Q_{XY}^{\otimes n}$, then $\sup_{P_{UXY}}\frac{I(U;Y^{n})}{I(U;X^{n})}$ can be strictly larger than $\rho^{2}$. Thus the cases of binary symmetric and Gaussian vectors cannot be proved via Lemma 1 as in the case of a pair of binary variables. This is a subtle issue that makes the proof of Theorem 5 somewhat nontrivial. Luckily, the symmetric strong data procesing constant tensorizes:

Lemma 7**.**

Let $(\mathbf{X,Y}):=(X_{j},Y_{j})_{j=1}^{n}\sim\otimes_{j=1}^{n}P_{X_{j}Y_{j}}$ for any given $P_{X_{j}Y_{j}}$, $j=1,\dots n$. Consider any random variables $W^{r}$, $r\in\mathbb{Z}$, satisfying

[TABLE]

*Then *

[TABLE]

where $P_{U^{r}|X_{j}Y_{j}}$ is such that

[TABLE]

Proof.

Note that by induction it suffices to consider $n=2$. Define

[TABLE]

Then note that the Markov chains

[TABLE]

are satisfied. Moreover,

[TABLE]

where we have defined $\Delta_{i}:=I(X_{1};Y_{2}|W^{i})-I(X_{1};Y_{2}|W^{i})$, and in (108) we have used the independence $Y_{1}\perp Y_{2}$ for the $i=1$ base case. Next, with similar algebra we obtain

[TABLE]

Then the claim (97) follows.

∎

Remark 10**.**

Above, we followed the original method of proof proposed in a classical paper of Kaspi [Kas85], which essentially builds on Csiszár-sum identity in multiuser information theory. This method has been used recently in testing for independence [XK13] and common randomness extraction [LCV17]. A similar method was applied in [BBCR10] to a problem of (approximately) reconstructing a function of two correlated iid strings.

9.3 Gaussian case

To obtain the same lower bound in the Gaussian case, we can use the result for binary symmetric sequence and apply a central limit theorem argument.

Lemma 8**.**

Let $X$ and $Y$ be jointly Gaussian with correlation $\rho$. Consider any random variables $U^{r}$, $r\in\mathbb{Z}$, $|U^{r}|<\infty$, satisfying

[TABLE]

Then $R(P_{U^{r}XY})\leq\rho^{2}S(P_{U^{r}XY})$.

Proof.

We claim the following continuity result: If $P^{(t)}_{XY}$ converges to $P_{XY}$ in the total variation distance, then $R(P^{(t)}_{U^{r}XY})$ and $S(P^{(t)}_{U^{r}XY})$ converge to $R(P_{U^{r}XY})$ and $S(P_{U^{r}XY})$ respectively, where $P^{(t)}_{U^{r}XY}:=P^{(t)}_{U^{r}|XY}P_{XY}$. The following will establish the lemma. Let $(A_{l},B_{l})_{l=1}^{t}$ be an iid sequence of binary symmetric random variables with correlation $\rho$, and put $X^{(t)}:=\frac{A_{1}+\dots+A_{t}}{\sqrt{t}}+a_{t}N$ and $Y^{(t)}:=\frac{B_{1}+\dots+B_{t}}{\sqrt{t}}+a_{t}N^{\prime}$, where $N$ and $N^{\prime}$ are standard Gaussian random variables, and $N$, $N^{\prime}$, $(X^{t},Y^{t})$ are independent. By the central limit theorem, we can choose some $a_{t}=o(1)$ such that the distribution of $(X^{(t)},Y^{(t)})$ converges to the Gaussian distribution $P_{XY}$ in total variation (Proposition 1). Now suppose that the claim is not true, then there exists $P_{U^{r}|XY}$ satisfying the required Markov chains and $|\mathcal{U}^{r}|<\infty$ such that $R(P_{U^{r}XY})>\rho^{2}S(P_{U^{r}XY})$. The continuity claim implies that $R(P^{(t)}_{U^{r}XY})>\rho^{2}S(P^{(t)}_{U^{r}XY})$ for some $t$. However, using the data processing inequality of mutual information it is easy to see that $R(P_{U^{r}A^{t}B^{t}})>R(P^{(t)}_{U^{r}XY})$ and that $S(P_{U^{r}A^{t}B^{t}})<S(P^{(t)}_{U^{r}XY})$. Thus $R(P_{U^{r}A^{t}B^{t}})>\rho^{2}S(P_{U^{r}A^{t}B^{t}})$, which is in contradition with Lemma 5 and Lemma 7.

It remains to prove the continuity claim. Note that for each $u^{r}$, $(x,y)\mapsto P_{U^{r}|XY}(u^{r}|x,y)$ is a measurable function taking values in $[0,1]$. Thus the convergence in total variation implies that $\lim_{t\to\infty}P_{U^{r}}^{(t)}(u^{r})=P_{U^{r}}(u^{r})$ and hence

[TABLE]

where the subscripts of $H$ denote the distributions with respect to which the entropies are computed. Moreover,

[TABLE]

is also a bounded measurable function, so

[TABLE]

and summing over $u^{r}$ shows that

[TABLE]

Note that (116) and $\eqref{e109}$ imply the convergence of $R(P_{U^{r}XY}^{(t)}):=I_{P^{(t)}}(U^{r};X,Y)$. Now,

[TABLE]

and hence it remains to show that

[TABLE]

By symmetry we only need to prove (120). Let us construct a coupling of $P_{U^{r}XY}$ and $P_{U^{r}XY}^{(t)}$ as follows. First construct a coupling such that $(X^{(t)},Y^{(t)})=(X,Y)$ with probability $\delta_{t}:=\frac{1}{2}|P_{XY}^{(t)}-P_{XY}|$. Let $E$ be the indicator of the event $(X^{(t)},Y^{(t)})\neq(X,Y)$. When $E=0$, generate $U^{r(t)}=U^{r}$ according to $P_{U^{r}|XY}(\cdot|X,Y)$. When $E=1$, generate $U^{r(t)}$ according to $P_{U^{r}|XY}(\cdot|X^{(r)},Y^{(r)})$ and $U^{r}$ according to $P_{U^{r}|XY}(\cdot|X,Y)$ independently. Then note that under either $P_{U^{r}XY}$ or $P^{(t)}_{U^{r}XY}$,

[TABLE]

Moreover,

[TABLE]

hence

[TABLE]

However, for any $\mathcal{A}\in\mathcal{X}$ and $u^{r}$,

[TABLE]

implying that $P^{(t)}_{U^{r}|X=x,E=0}(u^{r})=P_{U^{r}|X=x,E=0}(u^{r})$ for each $x$ and $u^{r}$, and hence $H_{P^{(t)}}(U^{r}|X,E=0)=H_{P}(U^{r}|X,E=0)$. Thus (122) and (125) imply that

[TABLE]

and (120) follows since $\delta_{t}\to 0$. ∎

Remark 11**.**

In [LCV17], we first computed the symmetric SDPI for the binary symmetric distribution and then proved the converse for secret key generation for the binary source. Then we proved a converse for the Gaussian source using the following reduction argument: Using a large block of binary symmetric random variables we can simulate a joint distribution converging to the Gaussian distribution in total variation. Thus the error probability of the operational problem cannot be too different under the simulated distribution and the true Gaussian distribution. In the present paper, we used a different argument to prove something stronger: the symmetric SPDI constant is equal to $\rho^{2}$ for the Gaussian distribution; this of course implies the converse for the operational problem.

10 Acknowledgments

We thank anonymous reviewer for the idea of reducing the general case to $(\rho,0)$, cf. Corollary 1. We note that this is precisely the method introduced by Rahman-Wagner [RW12], though they only used it in a one-way setting. We thank Yihong Wu, Mark Braverman, Rotem Oshman, Himanshu Tyagi and Sahasranand KR for useful discussions. This material is based upon work supported by the National Science Foundation CAREER award under grant agreement CCF-12-53205, the Center for Science of Information (CSoI), an NSF Science and Technology Center, under grant agreement CCF-09-39370, the European Research Council, under grant agreement 639573, the Israeli Science Foundation, under grant agreement 1367/14, the MIT IDSS Wiener Fellowship and the Yitzhak and Chaya Weinstein Research Institute for Signal Processing.