Inference under Information Constraints I: Lower Bounds from Chi-Square Contraction

TL;DR

This paper establishes lower bounds on the sample complexity for distribution learning and testing under information constraints, using chi-square contraction analysis, applicable to both privacy and communication limitations.

Contribution

It introduces a novel chi-square contraction framework and quantifies the impact of information constraints on sample complexity in distribution inference tasks.

Findings

Lower bounds are tight for public- and private-coin protocols.

Sample complexity for testing increases under private-coin constraints.

Characterization of chi-square contraction in local neighborhoods.

Abstract

Multiple players are each given one independent sample, about which they can only provide limited information to a central referee. Each player is allowed to describe its observed sample to the referee using a channel from a family of channels $\mathcal{W}$, which can be instantiated to capture both the communication- and privacy-constrained settings and beyond. The referee uses the messages from players to solve an inference problem for the unknown distribution that generated the samples. We derive lower bounds for sample complexity of learning and testing discrete distributions in this information-constrained setting. Underlying our bounds is a characterization of the contraction in chi-square distances between the observed distributions of the samples when information constraints are placed. This contraction is captured in a local neighborhood in terms of chi-square and decoupled chi-square fluctuations of a given channel, two quantities we introduce. The former captures the average distance between distributions of channel output for two product distributions on the input, and the latter for a product distribution and a mixture of product distribution on the input. Our bounds are tight for both public- and private-coin protocols. Interestingly, the sample complexity of testing is order-wise higher when restricted to private-coin protocols.

Tables

Table 1. TABLE I: Chi-square contraction lower bounds for local information-constrained learning and testing.

	Learning		Testing
	Public-Coin	Private-Coin	Public-Coin	Private-Coin
General	$\frac{k}{{\epsilon }^2}\cdot \frac{k}{{\max }_{W\in 𝒲}\parallel H\left(W\right))\parallel {}_{\ast }}$		$\frac{\sqrt{k}}{{\epsilon }^2}\cdot \frac{\sqrt{k}}{{\max }_{W\in 𝒲}{\parallel H\left(W\right)\parallel }_F}$	$\frac{\sqrt{k}}{{\epsilon }^2}\cdot \frac{k}{{\max }_{W\in \overline{𝒲}}{\parallel H\left(W\right)\parallel }_{\ast }}$
Communication	$\frac{k}{{\epsilon }^2}\cdot \frac{k}{2^{\ell }}$		$\frac{\sqrt{k}}{{\epsilon }^2}\cdot \sqrt{\frac{k}{2^{\ell }}}$	$\frac{\sqrt{k}}{{\epsilon }^2}\cdot \frac{k}{2^{\ell }}$
Privacy	$\frac{k}{{\epsilon }^2}\cdot \frac{k}{{\rho }^2}$		$\frac{\sqrt{k}}{{\epsilon }^2}\cdot \frac{\sqrt{k}}{{\rho }^2}$	$\frac{\sqrt{k}}{{\epsilon }^2}\cdot \frac{k}{{\rho }^2}$

Equations

$$\frac{W(y\mid x_1)}{W(y\mid x_2)}\le e^{\rho },\forall x_1,x_2\in [k],\forall y\in \{0,1{\}}^{\ast }.$$

$$\underset{\mathbf{p}\in \mathcal{P}}{\sup }\underset{X^n\sim \mathbf{p}}{\mathrm{Pr}}\left[d_{\mathrm{TV}}\left(\hat{\mathbf{p}}(X^n),\mathbf{p}\right)>\epsilon \right]<\frac{1}{12},$$

$$\underset{X^n\sim {\mathbf{p}}^n}{\mathrm{Pr}}\left[\mathcal{T}(X^n)=1\right]>\frac{11}{12}$$

$$\underset{X^n\sim {\mathbf{p}}^n}{\mathrm{Pr}}\left[\mathcal{T}(X^n)=0\right]>\frac{11}{12}$$

$$W\mathbf{p}(y):=\sum _x\mathbf{p}(x)W(y\mid x)={\mathbb{E}}_{\mathbf{p}}\left[W(y\mid X)\right];$$

$$H(W{)}_{i_1,i_2}:=\sum _{y\in \mathcal{Y}}\frac{(W(y\mid 2i_1-1)-W(y\mid 2i_1))(W(y\mid 2i_2-1)-W(y\mid 2i_2))}{{\sum }_{x\in [k]}W(y\mid x)},i_1,i_2\in [k/2].$$

$${\Delta }_k=\left\{\mathbf{p}:[k]\to [0,1]:{\Vert \mathbf{p}\Vert }_1=1\right\}.$$

$$d_{\mathrm{TV}}\left(\mathbf{p},\mathbf{q}\right):=\underset{S\subseteq \mathcal{X}}{\sup }\left(\mathbf{p}(S)-\mathbf{q}(S)\right)=\frac{1}{2}\sum _{x\in \mathcal{X}}\left|\mathbf{p}(x)-\mathbf{q}(x)\right|,$$

$${\mathrm{d}}_{{\chi }^2}\left(\mathbf{p},\mathbf{q}\right)$$

$$D\left(\mathbf{p}\Vert \mathbf{q}\right)$$

$$2\cdot {d_{\mathrm{TV}}\left(\mathbf{p},\mathbf{q}\right)}^2\le D\left(\mathbf{p}\Vert \mathbf{q}\right)\le {\mathrm{d}}_{{\chi }^2}\left(\mathbf{p},\mathbf{q}\right).$$

$$\Vert A{\Vert }_F={\left(\sum _{i=1}^m\sum _{j=1}^n{a}_{ij}^2\right)}^{1/2}={\left(\sum _{k=1}^{\min (m,n)}{\sigma }_k^2\right)}^{1/2}=\sqrt{\mathrm{Tr}{A}^{T}A}.$$

$$\Vert A{\Vert }_{\ast }:=\sum _{k=1}^{\min (m,n)}{\sigma }_k=\mathrm{Tr}\sqrt{A^{T}A},$$

$$\Vert A{\Vert }_F\le \Vert A{\Vert }_{\ast }\le \sqrt{\mathrm{rank}A}\cdot \Vert A{\Vert }_F,$$

$${\chi }^2(\mathcal{P}):=\frac{1}{|\mathcal{P}|}\sum _{\mathbf{p}\in \mathcal{P}}{\mathrm{d}}_{{\chi }^2}\left(\mathbf{p},\mathbf{q}\right),$$

$${\delta }_z(x):=\frac{{\mathbf{p}}_z(x)-\mathbf{q}(x)}{\mathbf{q}(x)},x\in [k],$$

$${\chi }^2(\mathcal{P})={\mathbb{E}}_Z\left[{\mathrm{d}}_{{\chi }^2}\left({\mathbf{p}}_Z,\mathbf{q}\right)\right]={\mathbb{E}}_Z\left[{\Vert {\delta }_Z\Vert }_2^2\right].$$

$${\chi }^{(2)}({\mathcal{P}}^n):=\log {\mathbb{E}}_{Z{Z}^{\prime }}\left[\exp \left(n\cdot ⟨{\delta }_Z,{\delta }_{Z^{\prime }}⟩\right)\right],$$

$$⟨{\delta }_z,{\delta }_{z^{\prime }}⟩:={\mathbb{E}}_{X\sim \mathbf{q}}\left[{\delta }_z(X){\delta }_{z^{\prime }}(X)\right]$$

$${\mathbf{p}}_z=\frac{1}{k}\left(1+2\epsilon z_1,1-2\epsilon z_1,\dots ,1+2\epsilon z_t,1-2\epsilon z_t,\dots ,1+2\epsilon z_{\frac{k}{2}},1-2\epsilon z_{\frac{k}{2}}\right),z\in \{-1,+1{\}}^{\frac{k}{2}}.$$

$${\delta }_z(x)=\{\begin{array}{ll}2\epsilon z_i,& x=2i-1,\\ -2\epsilon z_i,& x=2i,\end{array}i\in [k/2].$$

$${\chi }^2(\mathcal{P})=4{\epsilon }^2.$$

$$d_{\mathrm{TV}}\left({\mathbf{p}}_z,{\mathbf{p}}_{z^{\prime }}\right)=\mathrm{dist}\left(z,z^{\prime }\right)\cdot \frac{2\epsilon }{k},$$

$$\mathrm{Pr}\left[\mathrm{dist}\left(Z,\hat{Z}\right)>\frac{k}{6}\right]\le \frac{1}{12}.$$

$$\mathrm{Pr}\left[\mathrm{dist}\left(Z,\hat{Z}\right)>\frac{k}{6}\right]\ge 1-\frac{I(Z\wedge Y^n)+1}{{\log }_2|\mathcal{Z}|-{\log }_2{B}_{k/6}},$$

$${\log }_2{B}_{k/6}\le \frac{k}{2}\cdot h\left(\frac{1}{3}\right),$$

$$I(Z\wedge X^n)+1\ge \frac{11k}{12\cdot 2\cdot (1-h(1/3))}\ge \frac{k}{30}.$$

$$I(Z\wedge X^n)$$

$$={\mathbb{E}}_Z\left[D\left({\mathbf{p}}_Z^n\Vert {\mathbf{u}}^n\right)\right]-D\left({\mathbb{E}}_Z\left[{\mathbf{p}}_Z^n\right]\Vert {\mathbf{u}}^n\right)$$

$$\le {\mathbb{E}}_Z\left[D\left({\mathbf{p}}_Z^n\Vert {\mathbf{u}}^n\right)\right]$$

$$=n{\mathbb{E}}_Z\left[D\left({\mathbf{p}}_Z\Vert \mathbf{u}\right)\right]$$

$$\le n{\mathbb{E}}_Z\left[{\mathrm{d}}_{{\chi }^2}\left({\mathbf{p}}_Z,\mathbf{u}\right)\right]$$

$$=n\cdot {\chi }^2(\mathcal{P}),$$

$$\displaystyle\max_{z\in\mathcal{Z}}\bigg{|}\left\{z^{\prime}\in\mathcal{Z}:{\operatorname{d}_{\rm TV}\!\left({\mathbf{p}_{z},\mathbf{p}_{z^{\prime}}}\right)}\leq\frac{\varepsilon}{3}\right\}\bigg{|}\leq C_{\varepsilon}\,.$$

Full text

Underlying our bounds is a characterization of the contraction in chi-square distance between the observed distributions of the samples when information constraints are placed. This contraction is captured in a local neighborhood in terms of chi-square and decoupled chi-square fluctuations of a given channel, two quantities we introduce. The former captures the average distance between distributions of channel output for two product distributions on the input, and the latter for a product distribution and a mixture of product distribution on the input. Our bounds are tight for both public- and private-coin protocols. Interestingly, the sample complexity of testing is order-wise higher when restricted to private-coin protocols.

††footnotetext:

Jayadev Acharya is supported in part by the grant NSF-CCF-1846300 (CAREER), NSF-CCF-1815893, and a Google Faculty Fellowship. Part of this work was performed while Clément Canonne was supported by a Motwani Postdoctoral Fellowship at Stanford University. Himanshu Tyagi is supported in part by a research grant from the Robert Bosch Center for Cyberphysical Systems (RBCCPS), Indian Institute of Science, Bangalore.

1. I Introduction

2. I-A Information-constrained inference framework

3. I-B Summary of our techniques

4. I-C Prior and related work

5. I-D Organization

6. II Notation and preliminaries

7. III Perturbed families, chi-square fluctuations, and centralized lower bounds

8. III-A Chi-square fluctuation and the centralized learning lower bound

9. III-B Decoupled chi-square fluctuation and the centralized testing lower bound

10. IV Results: The chi-square contraction bounds with information constraints

11. IV-A General chi-square fluctuation bounds

12. IV-B Chi-square contraction bounds for learning and testing discrete distributions

13. V Examples and applications

14. V-A Communication-constrained inference

15. V-B Local differential privacy constraints

16. VI Future directions and upcoming results

17. A Proof of Section IV-B

18. B Proofs of chi-square fluctuation bounds

19. C Proofs of induced chi-square fluctuation bounds

I Introduction

Large-scale distributed inference has taken a center stage in many machine learning tasks. In these settings, it is becoming increasingly critical to operate under limited resources at each player, where the players (who hold the data samples) may be limited in their computational capabilities, communication capabilities, or may restrict the information about their data to maintain privacy. Our focus in this work will be on the last two constraints of communication and privacy, and, in general, on local information constraints on each player’s data.

We propose the following general framework for distributed statistical inference under local information constraints. There are $n$ players observing independent samples $X_{1},\dots,X_{n}$ from an unknown distribution $\mathbf{p}$ over a domain $\mathcal{X}$, with player $i$ getting the sample $X_{i}\in\mathcal{X}$. The players want to enable a central referee $\mathcal{R}$ to complete an inference task about their data. However, the players are constrained in the amount of information they can reveal to $\mathcal{R}$ about their observations in the following manner: Player $i$ must choose a channel $W_{i}$ from a prespecified class of channels $\mathcal{W}$ whose input alphabet is $\mathcal{X}$ and output alphabet is $\mathcal{Y}$, and use it to report its observed sample to $\mathcal{R}$.111A channel $W$ from $\mathcal{X}$ to $\mathcal{Y}$ is a randomized mapping $W\colon\mathcal{X}\to\mathcal{Y}$. We represent it by a $|\mathcal{Y}|\times|\mathcal{X}|$ transition probability matrix $W$ whose rows and columns are indexed by $y\in\mathcal{Y}$ and $x\in\mathcal{X}$, respectively, and its $(y,x)$th entry $W(y\mid x):=W_{y,x}$ is the probability of observing $y$ when the input to the channel is $x$. That is, player $i$ passes its observation $X_{i}$ as input to its chosen channel $W_{i}$ and $\mathcal{R}$ receives the channel’s output $Y_{i}$. The central referee then uses messages $Y_{1},\dots,Y_{n}$ from the players to complete the inference task of interest, such as estimation or testing for the underlying distribution $\mathbf{p}$; Fig. 1 illustrates the setup.

The family of allowed channels $\mathcal{W}$ serves as an abstraction of the local information constraints placed on each player’s message to $\mathcal{R}$. Before moving ahead, we instantiate this abstraction with two important examples, local communication constraints and local privacy constraints, and specify the corresponding $\mathcal{W}$s.

• (a)

Communication-Limited Inference. Each player can only send $\ell$ bits about their sample, i.e., player $i$ can send a message $Y_{i}\in\{0,1\}^{\ell}$. This constraint is captured by considering the set of allowed channels $\mathcal{W}=\mathcal{W}_{\ell}:=\{W\colon\mathcal{X}\to\{0,1\}^{\ell}\}$, the family of channels with output alphabet $\{0,1\}^{\ell}$.

• (b)

Locally Differentially Private Inference. Each player seeks to maintain privacy of their own data. We adopt the notion of local differential privacy which, loosely speaking, requires that no output message from a player reveals too much about its sample. This is captured by restricting $\mathcal{W}$ to $\mathcal{W}_{\rho}$, the family of $\rho$-locally differentially private ($\rho$-LDP) channels $W\colon\mathcal{X}\to\{0,1\}^{\ast}$ that satisfy the following (cf. [23, 38, 9, 21]): For $\rho>0$,

[TABLE]

These specific cases of communication and privacy constraints have received significant attention in the literature, and we emphasize these cases separately in our results. We emphasize, however, that our results are valid for arbitrary families $\mathcal{W}$ and can handle other examples from the literature such as the $t$-step Markov transition matrices considered in [10].

Our proposed framework and the general tools we develop are applicable to statistical inference for any family of distributions, with the caveat that deriving concrete results for a general family will require more work. Our focus in this work will be on discrete distributions, i.e., distributions on a finite alphabet $\mathcal{X}$. For this setting, we consider the canonical inference problems of estimating $\mathbf{p}$ and testing goodness-of-fit, under both communication and privacy constraints. Motivated by applications in distributed inference under resource constraints, we seek algorithms that enable the desired inference using the fewest samples possible, or equivalently, the least number of players. Our main results present a general approach for establishing lower bounds for the sample complexity of performing a given inference task under the aforementioned information-constrained setting. Underlying our lower bounds is a new quantitative characterization of contraction in the chi-square distance between two output message distributions, in comparison to that between the corresponding input distributions, as a function of the imposed information constraints represented by $\mathcal{W}$.

We allow randomized selection of $W$s from $\mathcal{W}$ at each player and distinguish between private-coin protocols, where this randomized selection is done independently for each player, and public-coin protocols, where the players can use shared randomness. Interestingly, our chi-square contraction bounds provide a quantitative separation of sample complexity for private-coin and public-coin protocols, an aspect hitherto ignored in the distributed inference literature and which is perhaps the main contribution of our work.

We summarize our results below, after a formal description of our problem setting.

I-A Information-constrained inference framework

We begin by recalling standard formulations for learning and testing discrete distributions in the classical non-distributed setting. Denote by $\Delta_{k}$ the set of all distributions over $[k]:=\{1,\ldots,k\}$. We set $\mathcal{X}$ to be $[k]$ and the set of unknown distributions to $\Delta_{k}$. Let $X^{n}:=(X_{1},\ldots,X_{n})$ be independent samples from an unknown distribution $\mathbf{p}\in\Delta_{k}$. We focus on the following two inference tasks.

Distribution Learning. In the $(k,\varepsilon)$-distribution learning problem, we seek to estimate a distribution $\mathbf{p}$ in $\Delta_{k}$ to within $\varepsilon$ in total variation distance. Formally, a (randomized) mapping $\hat{\mathbf{p}}\colon\mathcal{X}^{n}\to\Delta_{k}$ constitutes an $(n,\varepsilon)$-estimator if

[TABLE]

where $\operatorname{d}_{\rm TV}(\mathbf{p},\mathbf{q})$ denotes the total variation distance between $\mathbf{p}$ and $\mathbf{q}$ (see Section II for definition of total variation distance). Namely, $\hat{\mathbf{p}}$ estimates the input distribution $\mathbf{p}$ to within distance $\varepsilon$ with probability at least $11/12$. This choice of probability is arbitrary and has been chosen for convenience; see Footnote 12 to see where it is exactly used.

The sample complexity of $(k,\varepsilon)$-distribution learning is the least $n$ such that there exists an $(n,\varepsilon)$-estimator for $\mathbf{p}$. It is well-known that the sample complexity of distribution learning is $\Theta(k/\varepsilon^{2})$, and the empirical distribution attains it.

Identity Testing. In the $(k,\varepsilon)$-identity testing problem, given a known reference distribution $\mathbf{q}\in\mathcal{P}$, and samples from an unknown $\mathbf{p}$, we seek to test if $\mathbf{p}=\mathbf{q}$ or if it is $\varepsilon$-far from $\mathbf{q}$ in total variation distance. Specifically, an $(n,\varepsilon)$-test is given by a (randomized) mapping $\mathcal{T}\colon\mathcal{X}^{n}\to\{0,1\}$ such that

[TABLE]

In other words, upon observing independent samples $X^{n}$, the algorithm should “accept” with high constant probability if the samples come from the reference distribution $\mathbf{q}$ and “reject” with high constant probability if they come from a distribution significantly far from $\mathbf{q}$. Note again that the choice of $1/12$ for probability of error is for convenience.222In other words, we seek to solve the composite hypothesis testing problem with null hypothesis $\mathcal{H}_{0}=\{\mathbf{q}\}$ and composite alternative given by $\mathcal{H}_{1}=\left\{\;\mathbf{q}^{\prime}\in\Delta_{k}\;\colon\;{\operatorname{d}_{\rm TV}\!\left({\mathbf{q}^{\prime},\mathbf{q}}\right)}>\varepsilon\;\right\}$ in a minimax setting, with both type-I and type-II errors set to $1/12$.

The sample complexity of $(k,\varepsilon)$-identity testing is the least $n$ for which there exists an $(n,\varepsilon)$-test for $\mathbf{p}$. Clearly, this quantity can depend on the reference distribution $\mathbf{q}$. However, it is customary to consider the sample complexity over the worst-case $\mathbf{q}$.333The sample complexity for a fixed $\mathbf{q}$ has been studied under the “instance-optimal” setting (see [50, 12]): while the question is not fully resolved, nearly tight upper and lower bounds are known. In this worst-case setting, while it has been known for some time that the most stringent sample requirement arises for $\mathbf{q}$ set to the uniform distribution, a recent result of [29] provides a formal reduction of arbitrary $\mathbf{q}$ to the uniform distribution case. It is therefore enough to restrict $\mathbf{q}$ to being the uniform distribution; identity testing for the uniform reference distribution is termed the $(k,\varepsilon)$-uniformity testing problem. The sample complexity of $(k,\varepsilon)$-uniformity testing was shown to be \Theta\big{(}\sqrt{k}/\varepsilon^{2}\big{)} in [41].

We consider the two inference tasks above in our information-constrained setting. Let $\mathcal{W}$ be the set of allowed channels describing the constraints, and let as before $X_{1},\ldots,X_{n}$ be generated independently from an unknown distribution $\mathbf{p}\in\Delta_{k}$. Player $i$ chooses a channel $W_{i}\in\mathcal{W}$, passes its input $X_{i}$ through $W_{i}\in\mathcal{W}$ and the output message $Y_{i}$ constitutes information shared by player $i$ with $\mathcal{R}$. For a given choice of channel $W$ and $y\in\mathcal{Y}$, denote by $W\mathbf{p}$ the probability

[TABLE]

namely, $W\mathbf{p}$ is the distribution of the output message for a choice $W\in\mathcal{W}$ of the channel. The referee $\mathcal{R}$, upon observing the messages $Y^{n}$ from the players, seeks to solve the two inference tasks of $(k,\varepsilon)$-distribution estimation and $(k,\varepsilon)$-identity testing.

In choosing the channels $W$ from $\mathcal{W}$, the players can be allowed to follow protocols with different information structures. In the most general case, the choice $W_{i}$ of each player can depend on all the messages sent by the previous player and shared, public coins available to the players. In this work, we do not allow this most general class of protocols and restrict our attention to simultaneous message passing (SMP) protocols for communication. In an SMP protocol, the messages $Y_{1},\dots,Y_{n}$ from all players are transmitted simultaneously to the central server, and no other communication is allowed. However, we consider both the cases where public coins are and are not available. Note that this is equivalent to choosing the channels $W_{1},\dots,W_{n}$ simultaneously, possibly using public coins when they are available.

Note that the SMP setting forbids communication between the players, but does allow them to a priori agree on a strategy to select different mappings $W_{i}$ from $\mathcal{W}$. In this context, the role of shared randomness available to the players is important and motivates us to distinguish the settings of private-coin and public-coin protocols. In fact, as pointed-out earlier, a central theme of this work is to demonstrate the role of shared randomness available as public coins in enabling distributed inference. We show that it is indeed a resource that can greatly reduce the sample complexity of distributed inference.

Formally, the private- and public-coin SMP protocols are described as follows.

Definition \thedefn (Private-coin SMP Protocols).

Let $U_{1},\dots,U_{n}$ denote independent random variables, which are independent jointly of $(X_{1},\dots,X_{n})$.444In this work, we are not concerned with the amount of private or public randomness used. Thus, we can assume $U_{i}$s to be discrete random variables, distributed uniformly over a domain of sufficiently large cardinality. In a private-coin SMP protocol, player $i$ is given access to $U_{i}$ and the channel $W_{i}\in\mathcal{W}$ is chosen as a function of $U_{i}$. The central referee $\mathcal{R}$ does not have access to the realization of $U^{n}:=(U_{1},\dots,U_{n})$ used to generate the $W_{i}$s; however, it knows the mapping from $U_{i}$s to $W_{i}$s.

Definition \thedefn (Public-coin SMP Protocols).

Let $U$ be a random variable independent of $(X_{1},\dots,X_{n})$. In a public-coin SMP protocol, all players are given access to $U$, and they select their respective channels $W_{i}\in\mathcal{W}$ as a function of $U$. The central referee $\mathcal{R}$ is given access to the realization of $U$ as well, and its estimator and test can depend on $U$.

Note that in a private-coin SMP protocol, the channels $W_{1},\dots,W_{n}$ are independent since the $U_{i}$s are independent. Further, since $X_{i}$s are independent samples from $\mathbf{p}$, the messages $Y_{i}$s are also independent across the players. In particular, the distribution of $Y^{n}=(Y_{1},\ldots,Y_{n})$ is a product distribution. In contrast, in a public-coin SMP protocol, the channels $W_{i}$ (and hence $Y_{i}$s) are chosen as functions of the same random variable $U$ and therefore need not be independent. Nonetheless, even for public-coin SMP protocols, the messages $Y_{1},\ldots,Y_{n}$ are independent conditioned on the shared randomness $U$.

Remark I.1*.*

Throughout we assume that some randomness is available to generate the output of the channel $W_{i}$ given its input $X_{i}$. This randomness is assumed to be private as well (namely, it is independent across the players and is not available to $\mathcal{R}$). This assumption stands even for public-coin SMP protocols, implying the conditional independence of $Y_{i}$s given $U$ mentioned above, and is important in the context of privacy where the information available to $\mathcal{R}$ is seen as “leaked” and private randomness available only to the players is critical for enabling LDP channels.

We now define information-constrained discrete distribution estimation and testing at the referee.

For $(k,\varepsilon)$-distribution learning, an estimator using $\mathcal{W}$ comprises an SMP protocol that produces the messages $Y^{n}$ and an estimator mapping $\hat{p}$ that is applied by $\mathcal{R}$ to the messages $Y^{n}$. An $(n,\varepsilon)$-estimator using $\mathcal{W}$ is defined analogously to the centralized setting, by replacing the input $X^{n}$ of $\hat{p}$ with $(Y^{n},U)$ and $Y^{n}$, respectively, for public-coin and private-coin555It is important to note that $\mathcal{R}$ does not have access to the private randomness $U^{n}$. In fact, our lower bounds for private-coin protocols may not hold if the output at $\mathcal{R}$ can depend on private randomness of the players. SMP protocols. Similarly, for $(k,\varepsilon)$-identity testing, a test using $\mathcal{W}$ comprises an SMP protocol and a test mapping $\mathcal{T}$ applied by $\mathcal{R}$, and an $(n,\varepsilon)$-test using $\mathcal{W}$ is defined analogously to the centralized setting. We emphasize that the shared randomness used by the players (except that for realizing $W$) is available to $\mathcal{R}$, which only strengthens our lower bounds. Our main quantity of interest in this work is the following.

Definition \thedefn.

The sample complexity of $(k,\varepsilon)$-distribution learning or $(k,\varepsilon)$-identity testing (or $(k,\varepsilon)$-uniformity testing) using $\mathcal{W}$ for public-coin protocols, respectively, is the least $n$ such that there exists an $(n,\varepsilon)$-estimator or $(n,\varepsilon)$-test using $\mathcal{W}$ with a public-coin SMP protocol. The sample complexities of these tasks using $\mathcal{W}$ for private-coin protocols is defined analogously.

Since we are restricting to one sample per player, the sample complexity of these problems corresponds to the least number of players required to solve them as well. Our main objective in this line of work is the following:

Characterize the sample complexity for inference tasks using $\mathcal{W}$ for private- and public-coin protocols.

I-B Summary of our techniques

We are initiating a systematic study of the distributed inference problems described in the previous section. In this paper, the first in our series, we shall focus on lower bounds. For input distributions $\mathbf{p}$ and $\mathbf{q}$ over $\mathcal{X}$, the output messages are $W\mathbf{p}$ and $W\mathbf{q}$ over $\mathcal{Y}$. By data-processing inequalities, the output distributions $W\mathbf{p}$ and $W\mathbf{q}$ are “closer” than the corresponding input distributions $\mathbf{p}$ and $\mathbf{q}$, which makes it harder to perform inference using messages $Y_{i}$s than using the inputs $X_{i}$s themselves. We provide a quantitative characterization of this reduction in distance between the output distributions compared to the input distributions for the chi-square distance, which we term chi-square contraction, and use it to derive lower bounds for distributed inference problems.

In more detail, we study chi-square contractions for an $\varepsilon$-perturbed family (see Section III), a collection of probability distributions that are obtained by perturbing a nominal distribution. The perturbed family of distributions is chosen carefully to ensure that in order to accomplish the given inference task, an algorithm must roughly distinguish the perturbed distributions. In particular, we relate the difficulty of inference problems using two notions of distances: (a) the average chi-square distance between the perturbed distributions to the nominal distribution and (b) the chi-square distance of the average perturbed distribution to the nominal distribution. For our distributed inference setting, we need to bound these two quantities for the induced perturbed family of distributions at the outputs of the chosen channels from $\mathcal{W}$.

We provide bounds for these two quantities for channel output distributions in terms of two new measures of average distance in a neighborhood: the chi-square fluctuation for the average distance and the decoupled chi-square fluctuation for the distance to the average. The former notion has appeared earlier in the literature, albeit in different forms, and recovers known bounds for distributed distribution learning problems. The second quantity, the decoupled chi-square fluctuation, is the main technical tool introduced in this work, and leads to new lower bounds for distributed identity testing. Heuristically, the smaller these quantities are, the closer are the distributions of a perturbed family to the center, which in turn makes it harder to distinguish them and results in a higher sample complexity.

Observe that the general approach sketched above can be applied to any perturbed family. We obtain lower bounds for private-coin protocols by a maxmin evaluation of these bounds, where the maximum is over the choice of channels from $\mathcal{W}$ and the minimum is over the choice of perturbed families. In other words, since the channels are chosen independently of each other, for any given choice of channels, we will design a specific perturbed family to give rise to a small fluctuation bound. In contrast, we obtain lower bounds for public-coin protocols by a minmax evaluation of these bounds where the minimum is over perturbed families and the maximum is over the choice of channels from $\mathcal{W}$. In this case, we design a perturbed family such that for any choice of channels (chosen using the shared randomness), we can upper bound the chi-squared fluctuations. Remarkably, we establish that the maxmin evaluation can be significantly smaller than the minmax evaluation, leading to a quantitative separation in performance of private-coin and public-coin protocols for testing problems.

This separation has a heuristic appeal: On the one hand, in public-coin protocols players can use shared randomness to sample channels that best separate the current point in the alternative hypothesis class from the null. On the other hand, for a fixed private-coin protocol, one can identify a perturbed family in a “direction” where the current choice of channels will face difficulty in distinguishing the elements of the perturbed family. Further, we remark that this separation only holds for testing problems. This, too, makes sense in light of the previous heuristic since learning problems require us to distinguish all distributions in a neighborhood around the current hypothesis, without any preference for a particular “direction of perturbation.”

We develop these techniques systematically in Section III and Section IV. We begin by recasting the lower bounds for the standard, centralized, setting in our chi-square fluctuation language in Section III before extending these notions to the distributed setting in Section IV. Finally, we evaluate our general lower bounds for the distribution learning and identity testing problems.

Our lower bounds are obtained as a function of a channel-dependent matrix $H(W)$, defined below.

Definition \thedefn.

For any channel $W\in\mathcal{W}$, define $H(W)$ as the $k/2\times k/2$ positive semidefinite matrix given by:

[TABLE]

This matrix roughly captures the ability of the channel output to distinguish between even and odd inputs. Our bounds are in terms of the Frobenius norm $\lVert H(W){\rVert}_{F}$ and the nuclear norm $\lVert H(W){\rVert}_{\ast}$ of the matrix $H(W)$; see Section II for definitions. In effect, our results characterize the informativeness of a channel $W$ for distributed inference in terms of these norms of $H(W)$, and our final bounds for sample complexity involve the maximum of these norms over $W$ in $\mathcal{W}$. The smaller these norms are, the lower is the ability to distinguish consecutive inputs, and the better are our lower bounds (see Table I).

We summarize in Table I our sample complexity lower bounds for the $(k,\varepsilon)$-distribution learning and $(k,\varepsilon)$-identity testing problems for any general information constraints $\mathcal{W}$ for public- and private-coin protocols. The form here is only indicative; formal statements for results for general channels are available in Sections IV-B, IV-B and IV-B in Section IV and implications for specific $\mathcal{W}$ are given in Section V, with results on the communication-limited setting in Theorems V.1, V.2 and V.3 and the LDP setting in Theorems V.4, V.5 and V.6. The terms in each cell denotes the $\Omega(\cdot)$ lower bound obtained by our approach. The first row contains our lower bounds for a general family $\mathcal{W}$. We are specifying the bounds in terms of the multiplicative factor increase with respect to the central setting in which the sample complexity for learning and testing is $k/\varepsilon^{2}$ and $\sqrt{k}/\varepsilon^{2}$ respectively. We can instantiate the centralized setting by choosing $\mathcal{W}$ to contain the identity channel, which leads to $\lVert H(W){\rVert}_{\ast}=k$, and $\lVert H(W){\rVert}_{F}^{2}=k/2$ and retrieves the optimal bounds in the centralized setting.

As a corollary of these general bounds, we obtain $\Omega(k^{2}/(\varepsilon^{2}2^{\ell}))$ and $\Omega(k^{2}/(\varepsilon^{2}\rho^{2}))$ lower bounds for both private- and public-coin distribution learning using $\mathcal{W}_{\ell}$ (the communication-limited setting) and $\mathcal{W}_{\rho}$ (the LDP setting), respectively. In particular, the multiplicative increase is $k/2^{\ell}$ and $k/\rho^{2}$, respectively, for the communication-limited and LDP settings compared to the centralized setting. As discussed later, these bounds have also been obtained in previous works and are known to be tight.

We note that for communication-constrained identity testing, we obtain $\Omega(k/(\varepsilon^{2}2^{\ell/2}))$ and $\Omega(k^{3/2}/(\varepsilon^{2}2^{\ell}))$ lower bounds for public- and private-coin protocols respectively. Both these bounds are tight, thus establishing the first separation in sample complexity using public- and private-coin protocols for a natural distributed goodness-of-fit problem. In particular, when $\ell=1$ (one bit of communication per player) the sample complexity for public- and private-coin protocols is $\Theta(k)$ and $\Theta(k^{3/2})$ respectively. Similarly, for LDP identity testing, we obtain $\Omega(k/(\varepsilon^{2}\rho^{2}))$ and $\Omega(k^{3/2}/(\varepsilon^{2}\rho^{2}))$ lower bounds for public- and private-coin protocols, respectively, which are both tight, too, exhibiting the role of shared randomness in reducing the sample complexity.

As an interesting consequence, our results show that shared randomness does not help for distribution learning under communication or LDP constraints, in contrast to identity testing. Moreover, note that for $(k,\varepsilon)$-identity testing under general constraints, the factor $k/(\max_{W\in\mathcal{W}}\lVert H(W){\rVert}_{\ast})$ increase in the lower bound for private-coin protocols is the same as the increase for $(k,\varepsilon)$-distribution learning under information constraints; but the corresponding factor increase for identity testing using public-coin protocols is only $\sqrt{k}/(\max_{W\in\mathcal{W}}\lVert H(W){\rVert}_{F})$, which in general can be much smaller.

In the subsequent papers in this series ([3, 1]), we present public-coin and private-coin protocols to match the bounds in the communication-limited and LDP settings, respectively, thereby establishing the optimality of these lower bounds.

I-C Prior and related work

The statistical tasks of distribution learning and identity testing considered in this work have a rich history. The former requires no special techniques other than those used in parametric estimation problems with finite-dimensional parameter spaces, which are standard textbook material. The identity testing problem is the same as the classic goodness-of-fit problem. The latter goes beyond the discrete setting considered here, but often starts with a quantization to a uniform reference distribution (see [35, 39]). The focus in this line of research has always been on the relation of the performance to the support size (cf. [39]), with particular interest on the large-support and small-sample case where the usual normal approximations of statistics do not apply (cf. [40, 8]). Closer to our setting, Paninski [41] (see, also, [50]) established the sample complexity of uniformity testing, showing that it is sublinear in $k$ and equal to $\Theta(\sqrt{k}/\varepsilon^{2})$. As mentioned earlier, in this work we are following this sample complexity framework that has received attention in recent years. We refer the reader to surveys [18, 43, 15, 7] for a comprehensive review of recent results on discrete distribution learning and testing.

Distributed inference problems, too, have been studied extensively, although for the asymptotic, large-sample case and for simpler hypothesis classes. There are several threads here. Starting from Tsitsiklis [49], decentralized detection has received attention in the control and signal processing literature, with main focus on information structure, likelihood ratio tests and combining local decisions for global inference. In a parallel thread, distributed statistical inference under communication constraints was initially studied in the information theory community [5, 30, 31], with the objective to characterize the asymptotic error exponents as a function of the communication rate. Recent results in this area have focused on more complicated communication models [54, 53] and, more recently, on the minimum communication requirements for large sample sizes [44, 6].

Our focus is different from that of the works above. In our setting, independent samples are not available at one place, but instead information constraints are placed on individual samples. This is along the line of recent work on distributed mean estimation under communication constraints [58, 28, 45, 14, 55], although some of these works consider more general communication models than what we allow. The distribution estimation problem under communication constraints has been studied in [20],666To the best of our knowledge, only an extended abstract with the result statements is available, and a full version including the proofs is yet to appear. and, subsequent to an earlier version of this work, the distribution testing problem has been considered in [19]. However, in these two papers the authors consider a blackboard model of communication and strive to minimize the total number of bits communicated, without placing any restriction on the number of bits per sample. A more closely related variant of the distribution testing problem is studied in [26] where players observe multiple samples and communicate their local test results to the central referee who is required to use simple aggregation rules such as AND. Interestingly, such setups have received a lot of attention in the sensor network literature where a fusion center combines local decisions using simple rules such as majority; see [51] for an early review.

Closest to our work and independent of it is [32], which studies the $(k,\varepsilon)$-distribution learning problem using $\ell$ bits of communication per sample. It was shown that the sample complexity for this problem is $\Theta(k^{2}/(\varepsilon^{2}2^{\ell}))$. This paper in turn uses a general lower bound from [33, 34], which yields lower bounds for distributed parametric estimation under suitable smoothness conditions. For this special case, our general approach reduces to a similar procedure as [34], which was obtained independently of our work.

Distribution learning under LDP constraints has been studied in [21, 37, 56, 4, 52], all providing sample-optimal schemes with different merits. Our lower bound when specialized for this setting coincides with the one derived in [21].

In spite of this large body of literature closely related to our work, there are two distinguishing features of our approach. First, the methods for deriving lower bounds under local information constraints in all these works, while leading to tight bounds for distribution learning, do not extend to identity testing. In fact, our decoupled chi-square fluctuation bound fills this gap in the literature. We remark that distributed uniformity testing under LDP constraints has been studied recently in [46], however the lower bounds derived there are significantly weaker than what we obtain. Second, our approach allows us to prove a separation between the performances of public-coin and private-coin protocols. This qualitative lesson – namely that shared public randomness reduces the sample complexity – is in contrast to the prescription of [49] which showed that shared randomness does not help in distributed testing when the underlying problem is that of simple hypothesis testing.777As discussed earlier, identity testing is a composite hypothesis testing problem with null hypothesis $\mathbf{q}$ and alternative comprising all distributions $\mathbf{p}$ that are $\varepsilon$-far from $\mathbf{q}$ in total variation distance.

We observe that the unifying treatment based on chi-square distance is reminiscent of the lower bounds for learning under statistical queries (SQ) derived in [25, 24, 47]. On the one hand, the connection between these two problems can be expected based on the relation between LDP and SQ learning established in [38]. On the other hand, this line of work only characterizes sample complexity up to polynomial factors. In particular, it does not lead to lower bounds we obtain using our decoupled chi-square fluctuation bounds.

We close with a pointer to an interesting connection to the capacity of an arbitrary varying channel (AVC). At a high level, our minmax lower bound considers the worst perturbation for the best channel. This is semantically dual to the expression for capacity of an AVC with shared randomness, where the capacity is determined by the maxmin mutual information, with maximum over input distributions and minimum over channels (cf. [17]).

I-D Organization

We specify our notation in Section II and recall some basic inequalities needed for our analysis. This is followed by a review of the existing lower bounds for sample complexity of distribution learning and identity testing in Section III. In doing so, we introduce the notions of chi-square fluctuation which will be central to our work, and cast existing lower bounds under our general formulation. In Section IV, we generalize these notions to capture the information-constrained setting. Further, we apply our general approach to distribution learning and identity testing in the information-constrained setting. Then, in Section V, we instantiate these results to the settings of communication-limited and LDP inference and obtain our order-optimal bounds for testing and learning under these constraints. We conclude with pointers to schemes matching our lower bounds which will be reported in the subsequent papers in this series.

II Notation and preliminaries

Throughout this paper, we denote by $\log_{2}$ the logarithm to the base $2$ and by $\log$ the natural logarithm. We use standard asymptotic notation ${O\left(\cdot\right)}$, ${\Omega\left(\cdot\right)}$, and ${\Theta\left(\cdot\right)}$ for complexity orders.888Namely, for two non-negative sequences $(a_{n})_{n}$ and $(b_{n})_{n}$, we write $a_{n}=O(b_{n})$ (resp., $a_{n}=\Omega(b_{n})$) if there exist $C>0$ and $N\geq 0$ such that $a_{n}\leq Cb_{n}$ (resp., $a_{n}\geq Cb_{n}$) for all $n\geq N$. Further, we write $a_{n}=\Theta(b_{n})$ when both $a_{n}=O(b_{n})$ and $a_{n}=\Omega(b_{n})$ hold.

Let $[k]$ be the set of integers $\{1,2,\dots,k\}$. Given a fixed (and known) discrete domain $\mathcal{X}$ of cardinality $|\mathcal{X}|=k$, we write $\Delta_{k}$ for the set of probability distributions over $\mathcal{X}$, i.e.,

[TABLE]

For a discrete set $\mathcal{X}$, we denote by $\mathbf{u}_{\mathcal{X}}$ the uniform distribution on $\mathcal{X}$ and will omit the subscript when the domain is clear from context.

The total variation distance between two probability distributions $\mathbf{p},\mathbf{q}\in\Delta_{k}$ is defined as

[TABLE]

namely, ${\operatorname{d}_{\rm TV}\!\left({\mathbf{p},\mathbf{q}}\right)}$ is equal to half of the $\ell_{1}$ distance of $\mathbf{p}$ and $\mathbf{q}$. In addition to total variation distance, we will extensively rely on the chi-square distance $\operatorname{d_{\chi^{2}}}\left(\mathbf{p},\mathbf{q}\right)$ and Kullback–Leibler (KL) divergence ${D\left({\mathbf{p}\|\mathbf{q}}\right)}$ between distributions $\mathbf{p},\mathbf{q}\in\Delta_{k}$, defined as

[TABLE]

Using concavity of logarithms and Pinsker’s inequality, we can relate these two quantities to total variation distance as follows:

[TABLE]

In our results, we will rely on the following norms for matrices. Given a real-valued matrix $A=(a_{ij})_{(i,j)\in[m]\times[n]}$ with singular values $(\sigma_{k})_{1\leq k\leq\min(m,n)}$, the Frobenius norm (or Schatten 2-norm) of $A$ is given by

[TABLE]

Similarly, the nuclear norm (also known as trace or Schatten 1-norm) of $A$ is defined as

[TABLE]

where $\sqrt{A^{T}A}$ is the (principal) square root of the positive semi-definite matrix $A^{T}A$. For any $A$, the Frobenius and nuclear norms satisfy the following inequality

[TABLE]

which can be seen to follow, for instance, from an $\ell_{1}$/$\ell_{2}$ inequality and Cauchy–Schwarz inequality. Finally, the spectral radius of complex square matrix $A\in\mathbb{C}^{n\times n}$ with eigenvalues $\lambda_{1},\dots,\lambda_{n}$, is defined as $\rho(A):=\max_{1\leq i\leq n}\left\lvert\lambda_{i}\right\rvert$.

III Perturbed families, chi-square fluctuations, and centralized lower bounds

To build basic heuristics, we first revisit the derivation of lower bounds for sample complexity of $(k,\varepsilon)$-distribution learning and $(k,\varepsilon)$-identity testing in the centralized setting. As mentioned previously, for the latter it suffices to derive a lower bound for $(k,\varepsilon)$-uniformity testing. For brevity, we will sometimes refer to distribution learning as learning and identity testing as testing. We review both proofs in a unifying framework which we will extend to our information-constrained setting in the next section.999Although we restrict ourselves to the discrete distributions over $[k]$ here, the framework extends to more general parametric families.

Lower bounds for both learning and testing can be derived from a local view of the geometry of product distributions around the uniform distribution. Let $\mathbf{u}$ be the uniform distribution on $[k]$ and $\mathbf{u}^{n}$ be the $n$-fold product distributions over $[k]^{n}$, denoting the distribution of $n$ independent and identically distributed (i.i.d.) draws from $\mathbf{u}$. A typical lower bound proof involves designing an appropriate family of distributions close to $\mathbf{u}$ such that the underlying problem is information-theoretically difficult to solve even for this smaller family of distributions. We call such a family a perturbed family and define it next.

Definition \thedefn (Perturbed Family).

For $0<\varepsilon<1$ and a given $k$-ary distribution $\mathbf{q}$, an $\varepsilon$-perturbed family around $\mathbf{q}$ is a finite collection $\mathcal{P}$ of distributions such that, for all $\mathbf{p}\in\mathcal{P}$, ${\operatorname{d}_{\rm TV}\!\left({\mathbf{p},\mathbf{q}}\right)}\geq\varepsilon$.

When $\varepsilon$ is clear from context, we simply use the phrase perturbed family around $\mathbf{q}$.

As we shall see below, the bottleneck for learning distributions, which is a parametric estimation problem, arises from the difficulty in solving a multiple hypothesis testing problem with hypotheses given by the elements of a perturbed family around $\mathbf{u}$. Using Fano’s inequality, we can show that this difficulty is captured by the average KL divergence between $\mathbf{u}$ and the elements of the perturbed family. In fact, for a unified treatment, we shall simply bound KL divergences by chi-square distances. This motivates the following definition.

Definition \thedefn (Chi-square Fluctuation).

Given a perturbed family $\mathcal{P}$ around $\mathbf{q}$, the chi-square fluctuation of $\mathcal{P}$ is given by

[TABLE]

the average chi-square distance of the distributions in $\mathcal{P}$ from $\mathbf{q}$.

From Eq. 3, it follows that the average KL divergence mentioned above is upper bounded by the chi-square fluctuation of $\mathcal{P}$, which will be used to obtain a lower bound for the sample complexity of learning in the next section.

On the other hand, identity testing is a composite hypothesis testing problem, and a lower bound on the testing problem can be obtained using Le Cam’s two-point method. Specifically, for an $\varepsilon$-perturbed family $\mathcal{P}$ around $\mathbf{u}$, consider a binary hypothesis testing between the following two distributions over $[k]^{n}$: $\mathbf{u}^{n}$ and the uniform mixture $\frac{1}{|\mathcal{P}|}\sum_{\mathbf{p}\in\mathcal{P}}\mathbf{p}^{n}$ of the $n$-fold product of elements of the perturbed family. Since each element of $\mathcal{P}$ is at a total variation distance $\varepsilon$ from $\mathbf{u}$, it can be seen easily that any test for identity testing will yield a test for this binary hypothesis testing problem as well, with the same probability of error. In particular, a lower bound on the value of $n$ required to solve this problem gives a lower bound on identity testing. Thus, our goal is to capture the difficulty of this binary hypothesis testing problem. This difficulty is captured by the total variation distance between these two distributions on $[k]^{n}$, for which a simple upper bound is $\sqrt{n}\cdot\sqrt{\chi^{2}(\mathcal{P})}$. However, this bound turns out to be suboptimal.

Instead, an alternative bound derived using a recipe of Ingster [36] (the form here is from Pollard [42]) was shown to be tight in Paninski [41]. To understand this bound, we let the perturbed family $\mathcal{P}$ be parameterized by a discrete set $\mathcal{Z}$, i.e., for each $z\in\mathcal{Z}$, there is a $\mathbf{p}_{z}\in\mathcal{P}$. We will specify the choice of $\mathcal{Z}$ shortly. Denoting by $\delta_{z}\in\mathbb{R}^{k}$ the normalized perturbation with entries given by

[TABLE]

let ${\lVert\delta_{z}{\rVert}}_{2}^{2}:=\mathbb{E}_{X\sim\mathbf{q}}{\left[\delta_{z}(X)^{2}\right]}=\operatorname{d_{\chi^{2}}}\left(\mathbf{p}_{z},\mathbf{q}\right)$ be the second moment of the random variable $\delta_{z}(X)$ (for $X$ drawn from $\mathbf{q}$). For $Z$ uniform over $\mathcal{P}$, note that we can re-express $\chi^{2}(\mathcal{P})$ as

[TABLE]

Following [36, 42], we can bound the total variation distance mentioned above by a quantity we term the decoupled chi-square fluctuation of $\mathcal{P}$, instead of the weaker bound $\sqrt{n\cdot\chi^{2}(\mathcal{P})}$. This quantity appears by using the decoupling expression $\delta_{Z}^{2}=\delta_{Z}\delta_{Z^{\prime}}$, as will be seen below, and is defined next.

Definition \thedefn (Decoupled Chi-square Fluctuation).

Given a $k$-ary distribution $\mathbf{p}$ and a perturbed family $\mathcal{P}=\left\{\;\mathbf{p}_{z}\;\colon\;z\in\mathcal{Z}\;\right\}$ around $\mathbf{q}$, the $n$-fold decoupled chi-square fluctuation of $\mathcal{P}$ is given by

[TABLE]

where

[TABLE]

denotes the correlation inner product with respect to $\mathbf{q}$, and the outer expectation is over $Z$ and $Z^{\prime}$, which are independent and uniformly distributed uniformly over $\mathcal{Z}$.

While the quantities $n\cdot\chi^{2}(\mathcal{P})$ and $\chi^{(2)}(\mathcal{P}^{n})$ are implicit in prior work, the abstraction here allows us to have a clear geometric view and lends itself to the more general local information-constrained setting. For completeness, we review the proofs of existing lower bounds using our chi-square fluctuation terminology.

A specific perturbed family used in our lower bounds. In the sections below, we will present the proofs of lower bounds for sample complexity of learning and testing using a specific perturbed family $\mathcal{P}$ and bring out the role of $\chi^{2}(\mathcal{P})$ and $\chi^{(2)}(\mathcal{P}^{n})$ in these bounds. In particular, both bounds will be derived using the $\varepsilon$-perturbed family around $\mathbf{u}$ due to Paninski [41], consisting of distributions parameterized by $z\in\mathcal{Z}=\{-1,+1\}^{k/2}$ and comprising distributions $\mathbf{p}_{z}\in\Delta_{k}$ given by

[TABLE]

The normalized perturbations for this perturbed family are given by

[TABLE]

Note that for any $x\in[k]$, $|\delta_{z}(x)|=2\varepsilon$, and the chi-square fluctuation is given by

[TABLE]

III-A Chi-square fluctuation and the centralized learning lower bound

As a starting application, we recover the following well-known result for the sample complexity of distribution learning, using the notion of chi-squared fluctuation.

Theorem III.1**.**

For $(k,\varepsilon)$-distribution learning, if there exists an $(n,\varepsilon)$-estimator, then $n=\Omega(k/\varepsilon^{2})$.

To establish this bound, we consider the multiple hypotheses testing problem where the hypotheses are $\mathbf{p}_{z}$, $z\in\{-1,+1\}^{k/2}$, given in Eq. 7. Specifically, denote by $Z$ the random variable distributed uniformly on $\mathcal{Z}=\{-1,+1\}^{k/2}$ and by $X^{n}$ the random variable with distribution $\mathbf{p}_{Z}^{n}$ given $Z$. We can relate the accuracy of a probability estimate to the probability of error for the multiple hypothesis testing problem with hypotheses given by $\mathbf{p}_{z}$ using the standard Fano’s method (cf. [57]). In particular, we can use a probability estimate $\hat{\mathbf{p}}$ to solve the hypothesis testing problem by returning as $\hat{Z}$ a $z\in\{-1,1\}^{k/2}$ that minimizes ${\operatorname{d}_{\rm TV}\!\left({\mathbf{p}_{\hat{z}},\hat{\mathbf{p}}}\right)}$. The difficulty here is that the total variation distance ${\operatorname{d}_{\rm TV}\!\left({\mathbf{p}_{z},\mathbf{p}_{z^{\prime}}}\right)}$ may not be ${\Omega\left(\varepsilon\right)}$, and therefore, an $(n,\varepsilon)$-estimator may not return the correct hypothesis.

One way of circumventing this difficulty is to restrict to a perturbed family where pairwise-distances are ${\Omega\left(\varepsilon\right)}$. Note that for the perturbed family in Eq. 7

[TABLE]

where $\operatorname{dist}\left({z,z^{\prime}}\right)$ is the Hamming distance. This simple observation allows us to convert the problem of constructing a “packing” in total variation distance to that of constructing a packing in Hamming space. Indeed, a standard Gilbert–Varshamov construction of packing in Hamming space yields a subset $\mathcal{Z}_{0}\subset\{-1,+1\}^{k/2}$ with $|\mathcal{Z}_{0}|\geq 2^{ck}$ such that $\operatorname{dist}\left({z,z^{\prime}}\right)={\Omega\left(k\right)}$ for every $z,z^{\prime}$ in $\mathcal{Z}_{0}$. Using Fano’s inequality to bound the probability of error for this new perturbed family, we can relate the sample complexity of learning to $I(Z\wedge X^{n})$. However, when later extending our bounds to the information-constrained setting, this construction would create difficulties in bounding $I(Z\wedge X^{n})$ for public-coin protocols. We avoid this complication by relying instead on a slightly modified form of the classic Fano’s argument from [22]; this form of Fano’s argument was used in [34] as well to obtain a lower bound for the sample complexity of learning under communication constraints.

Specifically, in view of Eq. 10, it is easy to see that for an estimate $\hat{\mathbf{p}}$ such that $\mathbf{p}^{n}({\operatorname{d}_{\rm TV}\!\left({\mathbf{p},\hat{\mathbf{p}}}\right)}>\varepsilon/3)\leq 1/12$ for all $\mathbf{p}$, we must have

[TABLE]

On the other hand, the proof of Fano’s inequality in [16] can be extended easily to obtain (see, also, [22])

[TABLE]

where $B_{t}$ denotes the cardinality of Hamming ball of radius $t$. Noting that

[TABLE]

and combining the bounds above, if an $(n,\varepsilon)$-estimator exists, then we must have

[TABLE]

Therefore, to obtain a lower bound for sample complexity it suffices to bound $I(Z\wedge X^{n})$ from above. It is in this part that we bring in the role of chi-square fluctuations. Note that for a given value of $Z$, $X^{n}\sim\mathbf{p}_{Z}^{n}$. Therefore, we have

[TABLE]

where the last inequality uses ${D\left({\mathbf{p}\|\mathbf{q}}\right)}\leq\operatorname{d_{\chi^{2}}}\left(\mathbf{p},\mathbf{q}\right)$. Combining Eq. 13 and Eq. 14, we obtain that $n={\Omega\left(k/\chi^{2}(\mathcal{P})\right)}$, which along with Eq. 9 proves Theorem III.1.

In fact, the argument above is valid for any perturbation (i.e., around any nominal distribution $\mathbf{q}$) with the desired pairwise minimum total variation distance, namely any perturbed family satisfying an appropriate replacement for Eq. 12. In particular, it suffices to impose the following condition:

[TABLE]

Proceeding as in the proof of Theorem III.1 above, noting that $|\mathcal{Z}|=|\mathcal{P}|$ and replacing $B_{k/6}$ with the right-side of Eq. 15, we obtain the following.

Lemma \thelem.

For $0<\varepsilon<1$ and a $k$-ary distribution $\mathbf{q}$, let $\mathcal{P}$ be an $\varepsilon$-perturbed family around $\mathbf{q}$ satisfying Eq. 15. Then, the sample complexity of $(k,\varepsilon/3)$-distribution learning must be at least

[TABLE]

As a sanity check, when $\mathcal{P}$ is set to be Paninski’s perturbed family given in Eq. 7, we have $|\mathcal{P}|=2^{k/2}$, $C_{\varepsilon}=2^{(1-h(1/3))k/2}$, and $\chi^{2}(\mathcal{P})=4\varepsilon^{2}$ from Eq. 9, recovering the $\Omega(k/\varepsilon^{2})$ lower bound for sample complexity of distribution learning derived above.

III-B Decoupled chi-square fluctuation and the centralized testing lower bound

In this section, we provide an alternative proof for the following result on uniformity testing, using the notion of decoupled chi-square fluctuation.

Theorem III.2** ([41]).**

If there exists an $(n,\varepsilon)$-test for $(k,\varepsilon)$-uniformity testing, then $n=\Omega(\sqrt{k}/\varepsilon^{2})$.

Unlike for distribution learning, the binary hypothesis testing problems obtained from the pairs of distributions in the perturbed family $\mathcal{P}$ do not yield the desired dependence of sample complexity on $k$. As pointed earlier, the bottleneck in this case emerges by looking at a binary hypothesis testing problem between $\mathbf{u}^{n}$ and a uniform mixture of distributions from an $\varepsilon$-perturbed family. Specifically, by Le Cam’s method, any test for uniformity also constitutes a test for $\mathbf{u}^{n}$ versus $\mathbb{E}\!\left[\mathbf{p}_{Z}^{n}\right]=\frac{1}{2^{k/2}}\sum_{z\in\{-1,+1\}^{k/2}}\mathbf{p}_{z}^{n}$, the uniform mixture of $n$-fold product distributions over the perturbed family from Eq. 7. Thus, another aspect of the geometry around $\mathbf{u}^{n}$ that enters our consideration is the distance between $\mathbf{u}^{n}$ and $\mathbb{E}\!\left[\mathbf{p}_{Z}^{n}\right]$. In particular, the key would be to prove a lower bound on the value of $n$ to ensure that ${\operatorname{d}_{\rm TV}\!\left({\mathbf{u}^{n},\mathbb{E}\!\left[\mathbf{p}_{Z}^{n}\right]}\right)}$ is at least $1/12$.

As a straightforward attempt towards bounding this quantity, by Pinsker’s inequality and convexity of KL divergence, we get

[TABLE]

where the last identity is by Eq. 9. Thus, this upper bound on the distance between $\mathbf{u}^{n}$ and $\mathbb{E}\!\left[\mathbf{p}_{Z}^{n}\right]$ in terms of the chi-square fluctuation only yields a sample complexity lower bound of $\Omega(1/\varepsilon^{2})$, much lower than the $\Omega(\sqrt{k}/\varepsilon^{2})$ bound that we strive for.

Interestingly, we obtain the desired improvement in the lower bound by taking recourse to the decoupled chi-square fluctuation $\chi^{(2)}(\mathcal{P}^{n})$. Specifically, we have the following result.

Lemma \thelem.

For $0<\varepsilon<1$ and a $k$-ary distribution $\mathbf{q}$, let $\mathcal{P}$ be an $\varepsilon$-perturbed family around $\mathbf{q}$. Then, the sample complexity $n=n(k,\varepsilon)$ for $(k,\varepsilon)$-identity testing with reference distribution $\mathbf{q}$ must satisfy

[TABLE]

The proof is this result relies on the so-called Ingster’s method to bound the chi-square distance between a mixture of product distributions and a product distribution. The complete proof is provided in Appendix B, and makes critical use of the following result from [42], which, too, is proved in the appendix.

Lemma \thelem.

Consider a random variable $\theta$ such that for each $\theta=\vartheta$ the distribution $Q_{\vartheta}^{n}$ is defined as $Q_{1,\vartheta}\times\dots\times Q_{n,\vartheta}$. Further, let $P^{n}=P_{1}\times\dots\times P_{n}$ be a fixed product distribution. Then,

[TABLE]

where $\theta^{\prime}$ is an independent copy of $\theta$, and with $\delta_{j}^{\vartheta}(X_{j})=(Q_{j,\vartheta}(X_{j})-P_{j}(X_{j}))/P_{j}(X_{j})$,

[TABLE]

where the expectation is over $X_{j}$ distributed according to $P_{j}$.

Proof of Theorem III.2.

. We use Section III-B to complete the proof. Specifically, we apply the lemma to Paninski’s perturbed family given in Eq. 7. By Eq. 8,

[TABLE]

where $V_{i}:=Z_{i}Z_{i}^{\prime}$. Since $Z,Z^{\prime}$ are independently and uniformly distributed over $\{-1,+1\}^{k/2}$, $V_{1},\dots,V_{k/2}$ are independent and distributed uniformly over $\{-1,+1\}$. Therefore, we can bound the decoupled chi-square fluctuation using Hoeffding’s Lemma ($cf.$ [13]) as

[TABLE]

Thus, Section III-B implies that $\Omega(\sqrt{k}/\varepsilon^{2})$ samples are needed for testing (in particular, for uniformity testing). ∎

In closing, we summarize the geometry captured by the bounds derived in this section in Fig. 2. This geometry is a local view in the neighborhood of the uniform distribution obtained using the perturbed family $\mathcal{P}$ in Eq. 7. Each $\mathbf{p}_{z}$ is at a total variation distance $\varepsilon$ from $\mathbf{u}$. The mixture distribution we use is obtained by uniformly choosing the perturbation $\delta_{z}$ over $z\in\{-1,+1\}^{k/2}$.

The chi-square fluctuation of $\mathcal{P}$ is $O(n\varepsilon^{2})$ whereby the average total variation distance to $\mathbf{u}^{n}$ is $O(\sqrt{n}\varepsilon)$. On the other hand, the decoupled chi-square fluctuation of $\mathcal{P}$ is $O(n^{2}\varepsilon^{4}/k)$ and thus the total variation distance of the mixture of $\mathbf{p}_{z}^{n}$ to $\mathbf{u}^{n}$ is $O(n\varepsilon^{2}/\sqrt{k})$. Note that for $n\ll k/\varepsilon^{2}$, the total variation distance between the mixture $\mathbb{E}\!\left[\mathbf{p}_{Z}^{n}\right]$ and $\mathbf{u}^{n}$ is much smaller than the average total variation distance.

IV Results: The chi-square contraction bounds with information constraints

We now extend our notions of chi-square fluctuation and decoupled chi-square fluctuation to the information-constrained setting. We follow the same notation as the previous section. Recall that in the information-constrained setting each player sends information about its sample by choosing a channel from a family $\mathcal{W}$ to communicate to the central referee $\mathcal{R}$. The perturbed family will now induce a distribution on the outputs of the chosen channels $W_{1},\dots,W_{n}$. By data-processing inequalities these induced distributions will be closer to each other than the distributions of the perturbed family itself, and in particular the induced chi-squared fluctuations of the induced distributions will be smaller than that of the input distributions. We term this reduction in fluctuations chi-squared contraction. We will relate the difficulty of the learning and testing problems to the chi-squared fluctuations on the induced perturbed family. Combining these two steps we obtain lower bounds on the sample complexity of the learning and testing problems with respect to the centralized setting in terms of the chi-squared contractions. In other words, the difficulty of inference gets amplified by information constraints since the induced distributions are closer than the original ones and the chi-square fluctuation decreases.

We extend the general results from previous section to general information constraints $\mathcal{W}$ under both private-coin and public-coin protocols. In particular, we will extend Section III-A and Section III-B to the information-constrained setting. We then specialize them for discrete distribution learning and testing, *i.e.*for $\Delta_{k}$, by bounding the chi-squared fluctuations of the induced perturbed family corresponding to Paninski’s perturbed family of Eq. 7, for a given $\mathcal{W}$; and use them to obtain tight lower bounds for testing and learning under communication and privacy constraints for both private- and public-coin protocols. Underlying these bounds is a precise characterization of the contraction in chi-square fluctuation owing to information constraints. One can view this as a bound for the minmax chi-square fluctuation for an induced perturbed family, where the minimum is taken over perturbed families and the maximum over all channels in $\mathcal{W}$. We will see that for public-coin protocols, the bottleneck is indeed captured by this minmax chi-square fluctuation.

On the other hand, for private-coin protocols the bottleneck can be tightened further by designing a perturbation specifically for each choice of channels from $\mathcal{W}$. In other words, in this case we can use a bound for maxmin chi-square fluctuation. Another main result of this section, perhaps our most striking one, is a tight bound for this maxmin chi-square fluctuation for the aforementioned induced perturbed family. This bound turns out to be more stringent than the minmax chi-square fluctuation bound, and leads to the separation for private- and public-coin protocols for the cases $\mathcal{W}=\mathcal{W}_{\ell}$ and $\mathcal{W}=\mathcal{W}_{\rho}$ considered in the next section.

We begin by defining induced chi-squared fluctuations. Throughout we assume that the family of channels $\mathcal{W}$ consists of channels $W\colon\mathcal{X}\to\mathcal{Y}$ where the input alphabet is $\mathcal{X}$ and the output alphabet $\mathcal{Y}$ is finite. Recall from Eq. 1 that for an input distribution $\mathbf{p}$ over $\mathcal{X}$, the output distribution of channel $W$ is $\mathbf{p}(y):=\sum_{x}\mathbf{p}(x)W_{j}(y\mid x)=\mathbb{E}_{\mathbf{p}}{\left[W(y\mid X)\right]}.$

Let $\mathcal{P}$ be a perturbed family of distributions over $\mathcal{X}$ parameterized as $\left\{\;\mathbf{p}_{z}\;\colon\;z\in\mathcal{Z}\;\right\}$. As outlined above, our extension involves the notions of an induced perturbed family and its chi-square fluctuations, which is simply the family of distributions induced at the output for input distributions $\mathbf{p}_{z}$; formal definition follows.

Definition \thedefn.

For a perturbed family $\mathcal{P}$ and channels $W^{n}=(W_{1},\dots,W_{n})\in\mathcal{W}^{n}$, the induced perturbed family $\mathcal{P}^{W^{n}}$ comprises distributions $W^{n}\mathbf{p}_{z}^{n}$ over $\mathcal{Y}^{n}$ given by

[TABLE]

We now are able to define the notion of chi-square fluctuation from that of induced perturbed families. We first extend the corresponding notion of normalized perturbation with respect to the nominal distribution $\mathbf{q}$, defined in Eq. 5 as $\delta(x):=(\mathbf{p}(x)-\mathbf{q}(x))/\mathbf{q}(x)$. The induced normalized perturbation of $\mathbf{p}$ and $\mathbf{q}$ with respect to a channel $W$ is

[TABLE]

Thus, the normalized perturbation for the induced perturbed family $\mathcal{P}=\left\{\;\mathbf{p}_{z}\;\colon\;z\in\mathcal{Z}\;\right\}$ is given by

[TABLE]

where recall that $\delta_{z}(X)=(\mathbf{p}_{z}(x)-\mathbf{q}(x))/\mathbf{q}(x)$. As before, for a channel $W\in\mathcal{W}$, let ${\lVert\delta_{z}^{W}{\rVert}}_{2}^{2}:=\mathbb{E}_{Y\sim W\mathbf{q}}{\left[\delta_{z}^{W}(Y)^{2}\right]}=\operatorname{d_{\chi^{2}}}\left(W\mathbf{p}_{z},W\mathbf{q}\right)$.

Remark IV.1*.*

An important observation that will be used in our proofs later is that the random variable $\delta_{Z}^{W}$ can be obtained as a ($W$-dependent) linear transform of $\delta_{Z}$.

We now extend in the natural way the notions of chi-square fluctuations from the centralized setting from Sections III and III to the induced chi-squared fluctuations of $\mathcal{P}^{W^{n}}$.

Definition \thedefn.

Consider a perturbed family $\mathcal{P}=\{\mathbf{p}_{z}:z\in\mathcal{Z}\}$ around $\mathbf{q}$ and a family of channels $\mathcal{W}$. The induced chi-square fluctuation of $\mathcal{P}$ for $W\in\mathcal{W}$ is given by

[TABLE]

where $Z$ is distributed uniformly over $\mathcal{Z}$. The $n$-fold induced decoupled chi-square fluctuation of $\mathcal{P}$ for $W^{n}\in\mathcal{W}^{n}$ is given by

[TABLE]

where $\left\langle\delta_{z}^{W},\delta_{z^{\prime}}^{W}\right\rangle=\mathbb{E}_{Y\sim W\mathbf{q}}{\left[\delta_{z}^{W}(Y)\delta_{z^{\prime}}^{W}(Y)\right]}$.

We now make two relaxations to these notions. The first, to the definition of chi-squared fluctuations, to allow for more general $Z$; and the other to that of perturbed family, to weaken the requirement on distance from the nominal distribution. These relaxations will be helpful in investigating the role of shared randomness as is discussed later, in the paragraph before Section IV.

Our definitions until now have computed fluctuations by using a uniform distribution on the perturbed family $\mathcal{P}=\{\mathbf{p}_{z}:z\in\mathcal{Z}\}$. As can be seen from the results of the previous section, this is not required and all the results above extend to any distribution over $\mathcal{Z}$. We can consider a distribution $\zeta$ for $Z$, which need not even be independent across the coordinates $Z_{i}$s (when $Z$ is over $\{-1,+1\}^{k/2}$). For brevity, we will denote chi-square fluctuations for $\mathcal{P}$ when the expectation is computed using $\zeta$ by $\chi^{2}\left(W\mid\mathcal{P}_{\zeta}\right)$ and $\chi^{(2)}\left(W^{n}\mid\mathcal{P}_{\zeta}\right)$; when $\zeta$ is uniform, we omit the subscript $\zeta$ in $\mathcal{P}$.

In our definition of $\varepsilon$-perturbed family, we required ${\operatorname{d}_{\rm TV}\!\left({\mathbf{p}_{z},\mathbf{q}}\right)}$ to be bounded below by $\varepsilon$ for each $z\in\mathcal{Z}$. This requirement is imposed in view of Eq. 38 where it leads to the upper bound on probability of error. However, a nearly identical result can be obtained even if we relax this requirement to hold only with large probability. This motivates the next definition.

Definition \thedefn (Almost $\varepsilon$-Perturbation).

Fix $0<\varepsilon<1$, a family of distributions $\mathcal{P}=\{\mathbf{p}_{z},z\in\mathcal{Z}\}$, and a distribution $\zeta$ on $\mathcal{Z}$. The pair $\mathcal{P}_{\zeta}=(\mathcal{P},\zeta)$ is an almost $\varepsilon$-perturbation (around $\mathbf{q}$) if for some $\alpha\geq 1/10$

[TABLE]

where the probability is over $Z\sim\zeta$. We denote the set of all almost $\varepsilon$-perturbations by $\Upsilon_{\varepsilon}$ (We omit $\mathbf{q}$ from the notation as it will be clear from context).

The choice of $1/10$ in the definition above is used to match the probability of error requirement of $1/12$ in our PAC formulations given in Section I; see Eq. 41 and Footnote 12 for justification for these choices.

The flexibility offered by approximate perturbations is required to obtain our results for private-coin protocol; in particular, it will be used to show the separation between the performance of private- and public-coin protocols for testing. Recall that in Section III-B we showed that the existence of an identity test in the centralized setting implies that the chi-squared fluctuation must be bounded away from zero. We will establish an analogous result in the distributed setting, by defining two quantities which are the minmax and maxmin evaluations of the decoupled chi-squared fluctuation from Section IV. The two notions will be used to derive the lower bounds for information-constrained testing using public- and private-coin protocols, respectively.

Definition \thedefn (Minmax and Maxmin Chi-square Fluctuations).

For a family of channels $\mathcal{W}$, the $(n,\varepsilon)$-minmax decoupled chi-square fluctuation for $\mathcal{W}$ is given by

[TABLE]

and the $(n,\varepsilon)$-maxmin decoupled chi-square fluctuation for $\mathcal{W}$ is given by

[TABLE]

where the infimum is over all almost $\varepsilon$-perturbations $\mathcal{P}_{\zeta}$.

In Section IV-A, we extend the proofs of Section III-A and Section III-B to provide general results on learning and testing distributions under information constraints. Note that the desired extension to product distributions for Section IV-A requires Section III-B in its full generality (non-identical marginals for both $P^{n}$ and the $Q_{\theta}^{n}$s), in contrast to the earlier usage in the proof of Section III-B.

Further, we observe that when obtaining bounds for public-coin protocols we can restrict ourselves to a smaller family of channels than $\mathcal{W}$. The following notions are needed to state our results in full strength.

Definition \thedefn.

For a family of channels $\mathcal{W}$, denote by $\overline{\mathcal{W}}$ its convex hull, namely the set of channels $\overline{\mathcal{W}}=\left\{\;\theta W_{1}+(1-\theta)W_{2}\;\colon\;\theta\in[0,1],W_{1},W_{2}\in\mathcal{W}\;\right\}$. A generator family for $\mathcal{W}$, denoted $\mathcal{W}_{0}$, is a minimal subset of $\mathcal{W}$ whose convex hull is $\mathcal{W}$.

Note that the channels in $\mathcal{W}$ can be generated from and can generate, respectively, channels in $\mathcal{W}_{0}$ and $\overline{\mathcal{W}}$ using randomness.

IV-A General chi-square fluctuation bounds

The bounds presented in this section are obtained by relating the notions of chi-square fluctuation for $\mathcal{W}$ developed above to average distances in a neighborhood of the probability simplex. We present our bounds for learning and testing problems, but the recipe extends to many other inference problems. In the next section, we provide specific evaluations of these bounds for the Paninski perturbed family from Eq. 7, and some of its variants, which are tailored for the discrete distribution inference problems of learning and testing.

We begin with our bound for learning, which is a generalization of Section III-A to the information-constrained setting; the proof is provided in Appendix C.

**Lemma \thelem **(Chi-square fluctuation bound for

learning).

For $0<\varepsilon<1$ and a $k$-ary distribution $\mathbf{q}$, let $\mathcal{P}$ be an $\varepsilon$-perturbed family around $\mathbf{q}$ satisfying Eq. 15. Then, the sample complexity of $(k,\varepsilon)$-distribution learning using $\mathcal{W}$ for public-coin protocols is at least

[TABLE]

Note that the numerator is the same as in Section III-A which is the logarithm of a packing of distributions in total variation distance. The denominator, however, is now replaced with the induced chi-squared fluctuation.

The bounds above are for learning using public-coin protocols, and thus imply the same lower bound for learning with private-coin protocols. Interestingly, for testing, we obtain two different results. In Sections IV-A and IV-A we show two different conditions on the sample complexity of testing under public- and public-coin protocols in terms of the minmax and maxmin decoupled chi-squared fluctuations. We provide some insights into these bounds, whose proofs can be found in Appendix C. In the case of the maxmin decoupled chi-squared fluctuation, for any $W^{n}$ we maximize over all possible perturbations: this is the setting of private-coin protocols where after the channels are decided, we can design the perturbations to fool the channels. In contrast, for public-coin protocols, we must commit on a given perturbed family first, and then choose the best channels $W^{n}$. This is because the shared randomness can be leveraged to choose the set of channels to be dependent on each other.

**Lemma \thelem **(Minmax decoupled chi-square fluctuation bound for

testing).

For $0<\varepsilon<1$ and a $k$-ary reference distribution $\mathbf{p}$, the sample complexity $n=n(k,\varepsilon)$ of $(k,\varepsilon)$-identity testing using $\mathcal{W}$ for public-coin protocols must satisfy

[TABLE]

for some constant $c>0$ depending only on the probability of error.

Remark IV.2*.*

Using calculations similar to Eq. 17, we can obtain the following counterpart of Eq. 21: For every $\varepsilon$-perturbed family $\mathcal{P}$, it must hold that $\chi^{2}(\mathcal{W}_{0}^{n}\mid\mathcal{P})={\Omega\left(1\right)}$. Interestingly, even this bound, although seemingly as weak as Eq. 17, leads to useful bounds in the information-constrained setting. In particular, it will be seen in Section V to yield tight lower bounds for communication-constrained testing for $\ell=1$.

**Lemma \thelem **(Maxmin decoupled chi-square fluctuation bound for

testing).

For $0<\varepsilon<1$ and a $k$-ary reference distribution $\mathbf{p}$, the sample complexity $n=n(k,\varepsilon)$ of $(k,\varepsilon)$-identity testing using $\mathcal{W}$ for private-coin protocols must satisfy

[TABLE]

for some constant $c>0$ depending only on the probability of error.

IV-B Chi-square contraction bounds for learning and testing discrete distributions

All our main tools are in place. We now derive bounds for chi-square fluctuations for Paninski’s perturbed family of Eq. 7 and a related almost $\varepsilon$-perturbation, for arbitrary channel families $\mathcal{W}$. These bounds in turn will be used to obtain bounds for maxmin and minmax chi-square fluctuation. Combined with the chi-square fluctuation lower bounds of the previous section, these bounds yield concrete lower bounds on the sample complexity of learning and testing using $\mathcal{W}$. In essence, our bounds precisely characterize the contraction in chi-square fluctuation in the information-constrained setting over the standard setting; we term these bounds the chi-square contraction bounds.

As noted in Remark IV.1, the normalized perturbation $\delta_{Z}^{W}$ is linear in $\delta_{Z}$. Furthermore, for Paninski’s perturbed family, it follows from Eq. 5 that $\delta_{Z}$ itself is linear in $Z$. This observation allows us to capture chi-square fluctuations in terms of the channel-dependent $(k/2)\times(k/2)$ matrix101010Specifically, note that $H(W)$ is defined such that, for this perturbed family, $\left\langle\delta_{z}^{W},\delta_{z^{\prime}}^{W}\right\rangle=\frac{4\varepsilon^{2}}{k}z^{T}H(W)z^{\prime}$ for all $z,z^{\prime}$. $H(W)$ defined in Section I-B, whose expression we recall below:

[TABLE]

An important property of $H(W)$ that will be used throughout is that it is positive semi-definite. Indeed, we can express $H(W)$ as $\sum_{y}b_{y}b_{y}^{T}$ where the $b_{y}$s are $(k/2)$-length vectors with entries given by

[TABLE]

We are now in a position to state our main results. We start with a bound for chi-square fluctuation, which leads to a lower bound for sample complexity of learning.

Theorem IV.3**.**

For the $\varepsilon$-perturbed family $\mathcal{P}$ in Eq. 7 and any channel $W$, we have

[TABLE]

Remark IV.4*.*

A comparison of the bound above with Eq. 9 shows that the chi-square fluctuation contracts by a factor of at least $(4\max_{W\in\mathcal{W}}\,\lVert H(W){\rVert}_{\ast})/k$ due to the local information constraints corresponding to $\mathcal{W}$.

Before we prove this theorem, we show how to use it to obtain a lower bound for learning. Recalling Eq. 12, note that the perturbed family $\mathcal{P}$ given in Eq. 7 satisfies

[TABLE]

Thus, upon combining the chi-square fluctuation bound in Theorem IV.3 with Section IV-A, we obtain the following bound for the sample complexity of distribution learning.

Corollary \thecoro (Chi-square contraction bound for learning).

For $0<\varepsilon<1$, the sample complexity of $(k,\varepsilon)$-distribution learning using $\mathcal{W}$ for public-coin protocols is at least

[TABLE]

Proof of Theorem IV.3.

Using the expression of the normalized perturbation for $\mathcal{P}$ in Eq. 7, we get

[TABLE]

whereby

[TABLE]

where we have used the definition of $H(W)$ and the fact that $\mathbb{E}\!\left[Z_{i_{1}}Z_{i_{2}}\right]=\mathds{1}_{\left\{i_{1}=i_{2}\right\}}$. The claim follows upon noting that $\operatorname{Tr}{H(W)}=\lVert H(W){\rVert}_{\ast}$ since $H(W)$ is a positive semi-definite matrix. ∎

Next, we derive an upper bound for minmax chi-square fluctuation. As in the previous part, we obtain this bound by considering the perturbed family in Eq. 7.

Theorem IV.5**.**

Given $n\in\mathbb{N}$ and $\varepsilon\in(0,1)$, for a channel family $\mathcal{W}$ the minmax chi-square fluctuation is bounded as

[TABLE]

whenever

[TABLE]

Remark IV.6*.*

Comparing the bound above with Eq. 18 shows that the decoupled chi-square fluctuation contracts by a factor of $(\max_{W\in\mathcal{W}}\,\lVert H(W){\rVert}_{F}^{2})/k$ due to the local information constraints corresponding to $\mathcal{W}$ with respect to the centralized setting (where it was roughly $(n^{2}\varepsilon^{4}/k)$).

Before we prove the previous theorem, we note that combining the minmax decoupled chi-square fluctuation bound for testing of Section IV-A with Theorem IV.5 yields the following lower bound for the sample complexity of uniformity testing using public-coin protocols.

Corollary \thecoro (Chi-square contraction bound for testing using public-coin protocols).

For $0<\varepsilon<1$, the sample complexity of $(k,\varepsilon)$-uniformity testing using $\mathcal{W}$ for public-coin protocols is at least

[TABLE]

Proof of Theorem IV.5.

We consider the $\varepsilon$-perturbed family $\mathcal{P}$ defined in Eq. 7 and evaluate the fluctuation $\chi^{(2)}(\mathcal{P}^{n})=\log\mathbb{E}_{ZZ^{\prime}}{\left[\exp\left(n\cdot\left\langle\delta_{Z},\delta_{Z^{\prime}}\right\rangle\right)\right]}$ for this perturbed family.111111We need not invoke the more general notion of almost $\varepsilon$-perturbation for this proof; it suffices to use uniform distribution over an $\varepsilon$-perturbed family.

We apply Section III-B with $\vartheta=z$, $Q_{j,\vartheta}=W_{j}\mathbf{p}_{z}$, $P_{j}=W_{j}\mathbf{u}$, $1\leq j\leq n$ and $Z$ in the role of $\theta$. For brevity, denote by $\rho_{j,y}^{\mathbf{u}}$ and $\rho_{j,y}^{z}$, respectively, the probability that the output of player $j$ using channel $W_{j}$ is $y$ when the input distributions are $\mathbf{u}$ and $\mathbf{p}_{z}$. We have

[TABLE]

and that for every $z\in\{-1,1\}^{k/2}$,

[TABLE]

Therefore, the quantity $\delta_{j}^{Z}$ used in Section III-B is given by

[TABLE]

whereby for $1\leq j\leq n$ we get

[TABLE]

which upon substituting the expressions for $\rho_{j,y}$ and $\delta_{j}^{z}(y)$ from above yields

[TABLE]

where the matrix $H(W_{j})$ was introduced earlier in Eq. 2. Therefore,

[TABLE]

where we denote

[TABLE]

To prove the theorem, we need to bound the expression above in terms of the Frobenius norms of the matrices $H(W_{j})$. To that end, we use the following result on Rademacher chaos, whose proof is deferred to Appendix A.

Claim \theclm.

Let $\theta,\theta^{\prime}$ be two independent random vectors, each distributed uniformly over $\{-1,1\}^{k/2}$. Then, for any positive semi-definite matrix $H$,

[TABLE]

where $\lVert\cdot{\rVert}_{F}$ denotes the Frobenius norm and $\rho(\cdot)$ the spectral radius.

With this result at our disposal, we are ready to complete our proof. Setting $\lambda:=\frac{4n\varepsilon^{2}}{k}$, under assumption Eq. 25 we have

[TABLE]

where the second inequality uses convexity of norm. Rearranging the terms we obtain that $\lambda^{2}/(1-4\lambda^{2}\rho(\bar{H})^{2})\leq 4\lambda^{2}/3$, which when applied along with Section IV-B to Eq. 26 further yields

[TABLE]

where the penultimate inequality uses the convexity of $x^{2}$ in $x$; the proof is complete. ∎

Finally, we provide a bound for the maxmin chi-square fluctuation for a channel family $\mathcal{W}$.

Theorem IV.7**.**

Given $n\in\mathbb{N}$ and $\varepsilon\in(0,1)$, for a channel family $\mathcal{W}$ the $(n,\varepsilon)$-maxmin chi-square fluctuation is bounded as

[TABLE]

whenever

[TABLE]

where $C>0$ is a universal constant.

Remark IV.8*.*

Comparing the bound above with Eq. 18 shows that the decoupled chi-square fluctuation contracts by a factor of $(1/k^{2})\max_{W\in\mathcal{W}}\,\lVert H(W){\rVert}_{\ast}^{2}$ due to local information constraints, when restricting to private-coin protocols, which is worse than the contraction for public-coin protocols in view of Eq. 4.

Note that combining the maxmin decoupled chi-square fluctuation bound for testing in Section IV-A with Theorem IV.7 yields the following lower bound for the sample complexity of uniformity testing using private-coin protocols.

Corollary \thecoro (Chi-square contraction bound for testing using private-coin protocols).

For $0<\varepsilon<1$, the sample complexity of $(k,\varepsilon)$-uniformity testing using $\mathcal{W}$ for private-coin protocols is at least

[TABLE]

Before we provide a formal proof for Theorem IV.7, we summarize the high-level heuristics. In the proof of Theorem IV.5, we showed a bound for decoupled chi-square fluctuation of $\mathcal{P}$ for the induced perturbed family corresponding to the best choice of $W^{n}\in\mathcal{W}^{n}$. When only private-coin protocols are allowed, we can in fact design a perturbed family with the least decoupled chi-square fluctuation for the specific choice of $W^{n}$ used. Furthermore, we identify this least favorable direction of perturbation for $W^{n}$ by exploiting the spectrum of the positive semi-definite matrix $\bar{H}$ given in Eq. 27; details follow.

Proof of Theorem IV.7.

To obtain the desired bound for maxmin chi-square fluctuation, we derive a bound for decoupled chi-square fluctuation for an appropriately chosen almost $\varepsilon$-perturbation $\mathcal{P}_{\zeta}$. Specifically, consider a random variable $Z=(Z_{1},\dots,Z_{k/2})$ taking values in $[-1,1]^{k/2}$ and with distribution $\zeta$ such that for some constants $\alpha\geq 1/10$ and $c>0$,

[TABLE]

For $\varepsilon\in(0,c^{-1})$, consider the perturbed family around $\mathbf{u}$ consisting of elements $\mathbf{p}_{z}$, $z\in[-1,1]^{k/2}$, given by

[TABLE]

By the condition in Eq. 29 on $Z$, $\mathbf{p}_{Z}$ satisfies the following property with probability greater than $\alpha$:

[TABLE]

Note that if we set $Z_{i}=Y_{i}$ for $Y_{1},\dots,Y_{k/2}$ independent Rademacher random variables and the constant $c=2$, we recover the standard Paninski construction. However, we can do much more with this general construction. In particular, we can set $Z_{i}$s to be dependent, which will be used crucially in our proof. For a fixed channel family $\mathcal{W}$, we bound its $(n,\varepsilon)$-maxmin decoupled chi-square fluctuation by fixing an arbitrary $W^{n}\in\mathcal{W}^{n}$ and exhibit a perturbed family $\mathcal{P}_{\varepsilon}(\mathcal{W})=\mathcal{P}_{\zeta_{\mathcal{W}}}$ by designing a specific distribution $\zeta_{\mathcal{W}}$ to “fool” it.

We proceed by bounding $\chi^{(2)}\left(W^{n}\mid\mathcal{P}_{\zeta}\right)$ for a distribution $\zeta$ satisfying Eq. 29. Following the proof of Theorem IV.5, we get

[TABLE]

where $Z,Z^{\prime}$ are independent random variables with common distribution $\zeta$ and $H(W_{j})$ is defined as in Eq. 2. Note that

[TABLE]

where the matrix $\bar{H}$ is from Eq. 27. Informally, the matrix $\bar{H}$ captures the directions of the input space where the $n$-fold channel $W^{n}$ is the most informative; and thus, our goal is to design a distribution $\zeta$ which avoids these directions as much as possible.

To make this precise, let $0\leq\lambda_{1}\leq\lambda_{2}\leq\dots\leq\lambda_{k/2}$ be the eigenvalues of $\bar{H}$, and $\mathbf{v}^{1},\ldots,\mathbf{v}^{k/2}$ be corresponding (orthonormal) eigenvectors; in particular,

[TABLE]

Denote by $V$ the $(k/2)\times(k/4)$ matrix with columns given by $\mathbf{v}^{i}$ for $i\leq k/4$, i.e., the columns are the vectors corresponding to the $k/4$ smallest eigenvalues of $\bar{H}$. Let $Y_{1}\ldots Y_{k/4}$ be i.i.d. Rademacher random variables, and set $\zeta$ as the distribution of the random variable $Z:=VY$.

The first claim below shows that $\zeta$ satisfies Eq. 29.121212The probability guarantees obtained in Section IV-B determined our choice $1/12$ for the probability of error in our formulations in Section I.

Claim \theclm.

For $Z=VY$ described above, we have

[TABLE]

Proof.

For $m\in[k/2]$, we have $Z_{m}=\sum_{i=1}^{k/4}V_{m,i}Y_{i}$ where $V_{m,i}$ equals $\mathbf{v}^{i}_{m}$. Therefore, by Khintchine’s inequality ($cf.$ [48]),

[TABLE]

where $\mathbf{v}_{1},\dots,\mathbf{v}_{k/2}$ denote the row vectors of the matrix $V$.

Next, we note that ${\lVert\mathbf{v}_{m}{\rVert}}_{2}\leq 1$ for every $m\in[k/2]$. Indeed, denoting by $V^{\prime}$ the $(k/2)\times(k/2)$ matrix obtained by adding extra columns to $V$ to obtain a complete orthonormal basis for $\mathbb{R}^{k/2}$, we have $V^{\prime T}V^{\prime}=I$, whereby $V^{\prime}V^{\prime T}=I$. Thus, each row $\mathbf{v}_{m}^{\prime}$ of $V^{\prime}$ has ${\lVert\mathbf{v}_{m}^{\prime}{\rVert}}_{2}=1$, which gives

[TABLE]

Upon combining the bounds above, we obtain

[TABLE]

where in the second inequality we used ${\lVert\mathbf{v}_{m}{\rVert}}_{2}^{2}\leq 1$.

Moreover, note that

[TABLE]

which further gives

[TABLE]

Therefore, by the Paley–Zygmund inequality, for any $\theta\in(0,1)$

[TABLE]

The proof is completed by setting $\theta=1/3$. ∎

We will also require the following property, which is ensured by our construction of the matrix $V$.

Claim \theclm.

For $V\in\mathbb{R}^{(k/2)\times(k/4)}$ defined as above, we have

[TABLE]

Proof.

Note that since for $i_{1},i_{2}\in[k/4]$, we have

[TABLE]

Thus, by the orthonormality of $\mathbf{v}^{i}$s, the matrix $V^{T}\bar{H}V$ is diagonal, with diagonal entries $\lambda_{1},\dots,\lambda_{k/4}$. It follows that

[TABLE]

On the other hand, we also have

[TABLE]

and therefore,

[TABLE]

which is what we sought. ∎

We proceed to bound $\chi^{(2)}\left(W^{n}\mid\mathcal{P}_{\zeta}\right)$. First, note that

[TABLE]

where the first identity holds since $H(W)$ is positive semi-definite for every $W\in{\mathcal{W}}$. Using Section IV-B and the above bound, with the view of using Section IV-B and setting $\lambda:=(c^{2}n\varepsilon^{2})/k$, under assumption Eq. 28 (where $C=1/(8c^{2})$) we have

[TABLE]

Rearranging the terms to obtain $\lambda^{2}/(1-4\lambda^{2}\rho(\bar{H})^{2})\leq 4\lambda^{2}/3$ and applying Section IV-B to i.i.d. Rademacher random variables $Y$ and the symmetric matrix $V^{T}\bar{H}V\in\mathbb{R}^{k/4\times k/4}$ gives

[TABLE]

It remains to bound the Frobenius norm on the right-side above. To do so, we invoke once more Section IV-B which, along with Eq. 32, gives

[TABLE]

which completes the proof. ∎

On comparing Section IV-B and Section IV-B, we note that the effective contraction in decoupled chi-square fluctuation due to private-coin protocols is roughly $\frac{k}{\max_{W\in{\mathcal{W}}}\lVert H(W){\rVert}_{\ast}}$, which exceeds $\frac{\sqrt{k}}{\max_{W\in\mathcal{W}}\lVert H(W){\rVert}_{F}}$ for public-coin protocol since $H(W)$ has rank $O(k)$ and so by Eq. 4, $\lVert H(W){\rVert}_{\ast}\leq\sqrt{k}\cdot\lVert H(W){\rVert}_{F}$.

Remark IV.9*.*

Both channel families we consider in this paper, namely $\mathcal{W}_{\ell}$ for the communication-limited setting and $\mathcal{W}_{\rho}$ for the LDP setting, are convex and satisfy $\overline{\mathcal{W}}=\mathcal{W}$. Moreover, when evaluating bounds in Section IV-B and Section IV-B for these families, weaker bounds derived using $\mathcal{W}$ in place of $\mathcal{W}_{0}$ turn out to be optimal. Thus, our evaluations for these cases in the next section are based on $\mathcal{W}$ and do not require us to consider $\mathcal{W}_{0}$ or $\overline{\mathcal{W}}$. However, the more general form reported in this section may be useful elsewhere; in particular, in cases where one can identify a $\mathcal{W}_{0}$ that is more amenable to these bounds than $\mathcal{W}$ itself.

V Examples and applications

We now instantiate our general bounds for distribution learning and uniformity testing derived in the previous section to our two running examples of local information constraints, namely the communication-limited and LDP settings. We obtain tight lower bounds for sample complexity of learning and testing in these settings simply by bounding the Frobenius and trace norms of the associated matrices $H(W)$; see Table I for a summary of the results obtained. As mentioned earlier, we only focus on lower bounds here and delegate matching upper bounds to subsequent papers in this series.

V-A Communication-constrained inference

Recall that in the communication-limited setting, each player can transmit at most $\ell$ bits, which can be captured by using $\mathcal{W}=\mathcal{W}_{\ell}$, the family of channels from $[k]$ to $\mathcal{Y}=\{0,1\}^{\ell}$. To derive lower bounds for sample complexity of learning and testing for this case, Sections IV-B and IV-B require us to obtain upper bounds for $\max_{W\in\mathcal{W}_{0}}\lVert H(W){\rVert}_{\ast}$, $\max_{W\in\mathcal{W}_{0}}\lVert H(W){\rVert}_{\ast}$ and $\max_{W\in\overline{\mathcal{W}}}\lVert H(W){\rVert}_{\ast}$. We begin by observing that $\mathcal{W}$ is convex, whereby $\mathcal{W}=\overline{\mathcal{W}}$ which allows us to focus on $\lVert H(W){\rVert}_{\ast}$ and $\lVert H(W){\rVert}_{F}$ for $W\in\mathcal{W}$. Indeed, the convex combination of two $\ell$-bit output channels is an $\ell$-bit channel as well.

The next result provides bounds for the trace and Frobenius norms of the matrices $H(W)$ under communication constraints.

Lemma \thelem.

For a channel $W\colon[k]\to\{0,1\}^{\ell}$ and $H(W)$ as in Eq. 2, we have

[TABLE]

Proof.

Since matrix $H(W)$ is a positive semi-definite matrix, by the definition of nuclear norm in Section II, we have

[TABLE]

Moreover, for $y\in\mathcal{Y}$, denote by $\omega_{y}\in[0,1]^{[k/2]}$ the vector with the $i$th coordinate given by $\omega_{y,i}:=W(y\mid 2i-1)+W(y\mid 2i)$. Then,

[TABLE]

where in the penultimate identity we used the observation that $\sum_{y\in\mathcal{Y}}\omega_{y,i}=2$, for every $i\in[k/2]$. ∎

Plugging these bounds into Sections IV-B, IV-B and IV-B and recalling that $\mathcal{W}=\overline{\mathcal{W}}$ yield the following corollaries.

Theorem V.1** **(Communication-limited learning

using public coins).

The sample complexity of $(k,\varepsilon)$-distribution learning using $\mathcal{W}_{\ell}$ for public-coin protocols is at least ${\Omega\left(k^{2}/(2^{\ell}\varepsilon^{2})\right)}$.

Theorem V.2** (Communication-limited testing using public coins).**

The sample complexity of $(k,\varepsilon)$-uniformity testing using $\mathcal{W}_{\ell}$ for public-coin protocols is at least ${\Omega\left(k/(2^{\ell/2}\varepsilon^{2})\right)}$.

Theorem V.3** (Communication-limited testing using private coins).**

The sample complexity of $(k,\varepsilon)$-uniformity testing using $\mathcal{W}_{\ell}$ for private-coin protocols is at least ${\Omega\left(k^{3/2}/(2^{\ell}\varepsilon^{2})\right)}$.

Thus, the blow-up in sample complexity for communication-limited learning with public-coin protocols is a factor of $k/2^{\ell}$, which is the same for testing with private-coin protocols. This blow-up is reduced to a factor of $\sqrt{k/2^{\ell}}$ for testing with public-coin protocols. In fact, these bounds are tight and match the upper bounds in [3, 34] for learning, with a private-coin protocol achieving the public-coin lower bound, and [3] for both testing using private- and public-coin protocols.

V-B Local differential privacy constraints

Moving now to inference under LDP setting, recall that the information constraints here are captured by the family $\mathcal{W}_{\rho}$ of $\rho$-LDP channels $W\colon[k]\to\mathcal{Y}$ satisfying

[TABLE]

As before, we seek bounds for $\lVert H(W){\rVert}_{\ast}$ and $\lVert H(W){\rVert}_{F}$. Observe that $\mathcal{W}_{\rho}$ is convex: indeed, for $W_{1},W_{2}\in\mathcal{W}_{\rho}$, and for any $\theta\in[0,1]$, $\in\mathcal{W}$, and $i\neq j$,

[TABLE]

Thus, $\overline{\mathcal{W}_{\rho}}=\mathcal{W}_{\rho}$, and in the result below we may restrict to bounds for trace and Frobenius norms of $H(W)$ for $W\in\mathcal{W}_{\rho}$.

Lemma \thelem.

For $\rho\in(0,1]$, a $\rho$-LDP channel $W\in\mathcal{W}_{\rho}$ and $H(W)$ as in Eq. 2, we have

[TABLE]

Proof.

For the symmetric matrix $H(W)$ with $W\in\mathcal{W}_{\rho}$, we have

[TABLE]

where the first inequality as Eq. 33 implies that, for every $W\in\mathcal{W}_{\rho}$, $y\in\mathcal{Y}$, and $i_{1},i_{2},i_{3}\in[k]$,

[TABLE]

To see why, observe that when $W(y\mid i_{3})\geq W(y\mid i_{2})$, by Eq. 33 we have

[TABLE]

and when $W(y\mid i_{3})<W(y\mid i_{2})$ we have

[TABLE]

thereby establishing Eq. 34. Note that $\frac{1}{2}(e^{\rho}-1)^{2}=O(\rho^{2})$ for $\rho\in(0,1]$, which completes the proof of the bound for $\lVert H(W){\rVert}_{\ast}$. Moreover, from Eq. 4, we have $\lVert H(W){\rVert}_{F}^{2}\leq\lVert H(W){\rVert}_{\ast}^{2}=O(\rho^{4})$, concluding the proof of the lemma. ∎

Combining this with Sections IV-B, IV-B and IV-B, respectively, we obtain the following lower bounds on learning and testing under LDP constraints.

Theorem V.4** (LDP learning using public coins).**

For $\rho\in(0,1]$, the sample complexity $(k,\varepsilon)$-distribution learning using $\mathcal{W}_{\rho}$ for public-coin protocols is at least ${\Omega\left(k^{2}/(\rho^{2}\varepsilon^{2})\right)}$.

Theorem V.5** (LDP testing using public coins).**

For $\rho\in(0,1]$, the sample complexity of $(k,\varepsilon)$-uniformity testing using $\mathcal{W}_{\rho}$ for public-coin protocols is at least ${\Omega\left(k/(\rho^{2}\varepsilon^{2})\right)}$.

Theorem V.6** (LDP testing using private coins).**

For $\rho\in(0,1]$, the sample complexity of $(k,\varepsilon)$-uniformity testing using $\mathcal{W}_{\rho}$ for private-coin protocols is at least ${\Omega\left(k^{3/2}/(\rho^{2}\varepsilon^{2})\right)}$.

Similarly to the communication-limited setting, we see a separation between lower bounds for private- and public-coin protocols for testing under LDP constraints. In fact, the public-coin protocols for learning under LDP constraints from [21, 37, 56, 4, 52] match our lower bounds. Furthermore, [2, 1] provide private- and public-coin protocols for testing under LDP constraints that match our lower bounds here. Thus, indeed shared randomness strictly reduces sample complexity of testing when operating under LDP constraints.

VI Future directions and upcoming results

We have restricted our focus to lower bounds in this paper. Distributed inference schemes requiring number of players matching the lower bounds derived here will appear in two upcoming papers in this series. While these schemes will elaborate on the geometric view developed in this paper, the algorithms are new and tools needed for analysis are varied. We chose to organize these closely related papers into three separate parts for ease of presentation and to disentangle the distinct ideas involved.

In [3], the second paper in this series, we focus on the communication-constrained setting and provide public- and private-coin protocols for distributed inference whose performance matches the lower bounds presented here. A general strategy of “simulate-and-infer,” which is a private-coin protocol (and, in fact, a deterministic protocol), is used to achieve our bound learning as well as the bound for testing for private-coin protocols. On the other hand, a different scheme based on a random partition of inputs is used to attain bounds for testing with public-coin protocols. The efficacy of this latter scheme is closely tied to the geometric view developed here.

In [1], the third paper in this series, we provide schemes for testing under the LDP setting. For private-coin protocols, we simply use existing mechanisms such as RAPPOR and design sample-optimal tests for the $\mathcal{R}$. On the other hand, our bounds in this paper show that none of the existing LDP mechanisms, which are all private-coin protocols, can attain the public-coin lower bound. We present a new public-coin protocol that achieves our lower bounds here. Interestingly, our optimal public-coin protocol is similar to the one used in the communication-limited setting and draws on the geometric view developed here.

Finally, we point out that our framework readily extends to the high-dimensional and continuous settings, and can, for instance, be used to analyze the lower bounds for the problems of Gaussian mean testing and testing of product distributions under information constraints. We defer these interesting research directions to future work.

Appendix A Proof of Section IV-B

In this appendix, we prove Section IV-B which is recalled below for easy reference.

Claim \theclm (Section IV-B, restated).

Let $\theta,\theta^{\prime}$ be two independent random vectors, each distributed uniformly over $\{-1,1\}^{k/2}$. Then, for any positive semi-definite matrix $H$,

[TABLE]

where $\lVert\cdot{\rVert}_{F}$ denotes the Frobenius norm and $\rho(\cdot)$ the spectral radius.

Proof.

The proof follows closely that of [27, Proposition 8.13], which derives tail bounds on a homogeneous Rademacher chaos of order $2$ by bounding the moment-generating function. For $\theta,\theta^{\prime}$, and $H$ as above and $\lambda\in\mathbb{R}$,

[TABLE]

where to bound the inner expectation conditionally on $\theta$ we used the fact that Rademacher variables are sub-Gaussian and the sum of independent sub-Gaussian variables is sub-Gaussian. Since $H$ is symmetric, we can rewrite $\sum_{i_{1}=1}^{k/2}\left(\sum_{i_{2}=1}^{k/2}\theta_{i_{2}}H_{i_{1}i_{2}}\right)^{2}=\sum_{i_{2},i_{3}}\theta_{i_{2}}\theta_{i_{3}}\sum_{i_{1}}H_{i_{1}i_{2}}H_{i_{1}i_{3}}=\theta^{T}H^{2}\theta$. Thus, for $M:=H^{2}$ and $\mu\in\mathbb{R}$, we can consider

[TABLE]

where the first inequality is by the decoupling inequality $\mathbb{E}{\left[e^{\theta^{T}M\theta}\right]}\leq\mathbb{E}{\left[e^{\theta^{T}M\theta^{\prime}}\right]}$ (used in [27] as well) and the second uses sub-Gaussianity once again. Since $M=H^{T}H$ is positive semidefinite, we can rewrite

[TABLE]

where $\lVert M{\rVert}_{2}:=\sup_{{\lVert\mathbf{x}{\rVert}}_{2}\leq 1}\left\langle M\mathbf{x},\mathbf{x}\right\rangle$ is the operator norm of $M$. For $8\mu\lVert M{\rVert}_{2}\leq 1$, applying Jensen’s inequality to the concave function $t\mapsto t^{8\mu\lVert M{\rVert}_{2}}$ we get

[TABLE]

which yields

[TABLE]

Recalling that $\operatorname{Tr}M=\operatorname{Tr}(H^{2})=\lVert H{\rVert}_{F}^{2}$ and $\lVert M{\rVert}_{2}=\lVert H^{2}{\rVert}_{2}=\rho(H)^{2}$, and choosing $\mu=\lambda^{2}/2$ (which satisfies $8\mu\lVert M{\rVert}_{2}\leq 1$), we get from Eqs. 35 and 36 that

[TABLE]

which completes the proof. ∎

Appendix B Proofs of chi-square fluctuation bounds

Proof of Section III-B.

The proof uses Le Cam’s two-point method. We note first that

[TABLE]

and bound the right-side further using Section III-B with $\theta$ replaced by $z$, $Q_{\vartheta}^{n}=\mathbf{p}_{z}^{n}$, and $P_{i}=\mathbf{p}$ to get

[TABLE]

since $H_{1}(Z,Z^{\prime})=\left\langle\delta_{Z},\delta_{Z^{\prime}}\right\rangle$. Now, to complete the proof, consider an $(n,\varepsilon)$-test $\mathcal{T}$. By definition, we have $\Pr_{X^{n}\sim\mathbf{p}^{n}}\left[\,\mathcal{T}(X^{n})=1\,\right]>11/12$ and $\Pr_{X^{n}\sim\mathbf{p}_{z}^{n}}\left[\,\mathcal{T}(X^{n})=1\,\right]>11/12$ for every $z$, whereby

[TABLE]

The left-hand-side above coincides with the Bayes error for test $\mathcal{T}$ for the simple binary hypothesis testing problem of $\mathbb{E}\!\left[\mathbf{p}_{Z}^{n}\right]$ versus $\mathbf{p}^{n}$, which must be at least

[TABLE]

Thus, we obtain ${\operatorname{d}_{\rm TV}\!\left({\mathbb{E}\!\left[\mathbf{p}_{Z}^{n}\right],\mathbf{p}^{n}}\right)}\geq 5/6$, which together with Eq. 37 completes the proof. ∎

Proof of Section III-B.

Using the definition of chi-square distance, we have

[TABLE]

where the outer expectation is for $X^{n}$ using the distribution $P^{n}$. For brevity, denote by $\Delta_{i}^{\vartheta}$ the random variable $\delta_{i}^{\vartheta}(X_{i})$. The product in the expression above can be expanded as

[TABLE]

whereby we get

[TABLE]

Observe now that $\mathbb{E}_{P^{n}}{\left[\Delta_{i}^{\vartheta}\right]}=0$ for every $\vartheta$. Furthermore, $\theta^{\prime}$ is an independent copy of $\theta$ and $\Delta_{i}^{\theta}$ and $\Delta_{j}^{\theta^{\prime}}$ are independent for $i\neq j$. Therefore, the expectation on the right-side above equals

[TABLE]

which completes the proof. ∎

Appendix C Proofs of induced chi-square fluctuation bounds

Proof of Section IV-A.

The proof is nearly identical to that of Section III-A, with few additional observations. Using Fano’s inequality Eq. 11 and following the proof of Section III-A, it suffices to derive the counterpart of Eq. 14. Note that by definition of $\mathcal{W}_{0}$, any public-coin protocol can be realized by using a shared randomness $U$, together with $W_{1},\dots,W_{n}$ from $\mathcal{W}_{0}$. Thus, considering observations $(Y^{n},U)$ and proceeding as in Eq. 14,

[TABLE]

which completes the proof together with Eq. 11. ∎

Proof of Section IV-A.

Consider an almost $\varepsilon$-perturbation $\mathcal{P}_{\zeta}$. The proof of this extension is very similar to the proof of Section III-B, except that $\mathbb{E}\!\left[\mathbf{p}_{Z}^{n}\right]$ and $\mathbf{p}^{n}$ get replaced with $\mathbb{E}\!\left[W^{n}\mathbf{p}_{Z}^{n}\right]$ and $W^{n}\mathbf{p}^{n}$, respectively. The first part of the argument goes through verbatim, leading to

[TABLE]

for every choice of channels $W^{n}=(W_{1},\dots,W_{n})$. In the second step, we need to get a lower bound on the left-side above, while restricting to $W_{i}$s in $\mathcal{W}_{0}$. Towards that, consider an $(n,\varepsilon)$-test $\mathcal{T}$ using a public-coin protocol. Denoting by $U$ the public randomness and by $Y_{1},\dots,Y_{n}$ the messages from each player and by $\mathcal{Z}_{0}$ the set of $z$ such that ${\operatorname{d}_{\rm TV}\!\left({\mathbf{p}_{z},\mathbf{p}}\right)}\geq\varepsilon$. Since $\mathcal{P}_{\zeta}$ is an almost $\varepsilon$-perturbation, $\Pr\left[Z\in\mathcal{Z}_{0}\right]\geq\alpha\geq 1/10$. Also, for the test $\mathcal{T}$ we have $\Pr_{X^{n}\sim\mathbf{p}^{n}}\left[\,\mathcal{T}(U,Y^{n})=1\,\right]\geq 11/12$ and $\Pr_{X^{n}\sim\mathbf{p}_{z}^{n}}\left[\,\mathcal{T}(U,Y^{n})=1\,\right]\geq 11/12$ for every $z\in\mathcal{Z}_{0}$. Thus, in the manner of Eq. 38 we obtain

[TABLE]

where in the last inequality we used $\alpha\geq 1/10$. Then, we can find a fixed realization $U=u$ such that

[TABLE]

An important remark here is that $u$ may depend on $\mathcal{P}_{\zeta}$. Observe that by definition of $\mathcal{W}_{0}$, we can emulate the public-coin protocols by each player selecting its channel $W_{i}\in\mathcal{W}_{0}$ as a function of the shared randomness $U$. Denote by $W_{u}^{n}\in\mathcal{W}_{0}^{n}$ the channels chosen by the players when $U=u$. Then, conditioned on $U=u$, $Y^{n}$ has distribution $W_{u}^{n}\mathbf{p}^{n}$ and $W_{u}^{n}\mathbf{p}_{z}^{n}$, respectively, when $X^{n}$ has distribution $\mathbf{p}^{n}$ and $\mathbf{p}_{z}^{n}$. Thus, as in the proof of Section III-B, we can find $W_{u}^{n}\in\mathcal{W}_{0}^{n}$ such that

[TABLE]

which along with Eq. 39 yields

[TABLE]

where $c=\log(14401/14400)$. The result follows upon taking the maximum over $W_{u}^{n}\in\mathcal{W}_{0}^{n}$ and minimum over all almost $\varepsilon$-perturbations $\mathcal{P}_{\zeta}$. ∎

Proof of Section IV-A.

The argument follows the same template as the proof of Section IV-A, but with an important difference. Instead of derandomizing as in Eq. 41, which leads to a choice of channels $W^{n}_{u}$ that may depend on perturbation $\mathcal{P}_{\zeta}$ family, now in Eq. 42 we would like to take the minimum over $\mathcal{P}_{\zeta}\in\Upsilon_{\varepsilon}$ first. Observe that for private-coin protocols, the effective channel used by each player is a convex combination of channels from $\mathcal{W}$, namely it is a channel from $\overline{\mathcal{W}}$. Thus, when $X^{n}$ has distribution either $\mathbf{p}^{n}$ and $\mathbf{p}_{z}^{n}$, respectively, $Y^{n}$ has distribution $W^{n}\mathbf{p}^{n}$ and $W^{n}\mathbf{p}_{z}^{n}$ with $W^{n}\in\overline{\mathcal{W}}^{n}$. Therefore, following the steps in the proof of Section IV-A, we get $\chi^{(2)}\left(W^{n}\mid\mathcal{P}_{\zeta}\right)\geq c$, where $W^{n}\in\overline{\mathcal{W}}^{n}$ and the almost $\varepsilon$-perturbation $\mathcal{P}_{\zeta}$ is arbitrary. The claim then follows by taking the minimum over $\mathcal{P}_{\varepsilon}$ and maximum over $W^{n}\in\overline{\mathcal{W}}^{n}$. ∎

Acknowledgments

We thank Gautam Kamath and Vitaly Feldman for useful pointers to related work and Ayfer Özgür for sharing with us a draft of [34]. We thanks Lekshmi Ramesh for comments on an early draft and Prathamesh Mayekar for pointing a typo in an earlier version of Section I-A. We are also grateful to Mayank Bakshi for pointing out the connections to the AVC setting, and to Chandra Nair for an illuminating discussion on data processing inequalities for chi-square distances. Finally, we would like to thank the organizers of the 2018 Information Theory and Applications Workshop (ITA), where the collaboration leading to this work started.