The Role of Interactivity in Local Differential Privacy

TL;DR

This paper investigates the power of interactivity in local differential privacy, demonstrating that fully interactive protocols can be more sample-efficient than sequentially interactive ones, but noninteractive tests are often optimal for hypothesis testing.

Contribution

It classifies locally private protocols by compositionality, provides a transformation from fully to sequentially interactive protocols, and establishes tight bounds on their relative power.

Findings

Fully interactive protocols can be exponentially more sample-efficient than sequential ones.

Noninteractive hypothesis tests are optimal for a broad class of problems.

The paper provides tight bounds and transformations between different protocol types.

Abstract

We study the power of interactivity in local differential privacy. First, we focus on the difference between fully interactive and sequentially interactive protocols. Sequentially interactive protocols may query users adaptively in sequence, but they cannot return to previously queried users. The vast majority of existing lower bounds for local differential privacy apply only to sequentially interactive protocols, and before this paper it was not known whether fully interactive protocols were more powerful. We resolve this question. First, we classify locally private protocols by their compositionality, the multiplicative factor $k \geq 1$ by which the sum of a protocol's single-round privacy parameters exceeds its overall privacy guarantee. We then show how to efficiently transform any fully interactive $k$-compositional protocol into an equivalent sequentially interactive protocol with an $O(k)$ blowup in sample complexity. Next, we show that our reduction is tight by exhibiting a family of problems such that for any $k$, there is a fully interactive $k$-compositional protocol which solves the problem, while no sequentially interactive protocol can solve the problem without at least an $\tilde \Omega(k)$ factor more examples. We then turn our attention to hypothesis testing problems. We show that for a large class of compound hypothesis testing problems --- which include all simple hypothesis testing problems as a special case --- a simple noninteractive test is optimal among the class of all (possibly fully interactive) tests.

Equations

$${\mathbb{P}}_{\mathcal{M}}\left[\mathcal{A}(S)\in \Omega \right]\le \exp (\epsilon ){\mathbb{P}}_{\mathcal{M}}\left[\mathcal{A}(S^{\prime })\in \Omega \right]+\delta .$$

$$\sum _{t:i_t=i}{\epsilon }_t\le k\epsilon .$$

$$\sum _t{\epsilon }_t\le k\epsilon n.$$

$${\Pi }^f\stackrel{d}{=}{\Pi }^b$$

$$Y_{t+1}|i^{t+1},R_{t+1},{\epsilon }_{t+1},{\delta }_{t+1},{\Pi }_{<t+1}\sim R_{t+1}(x_{i^{t+1}},{\epsilon }_{t+1},{\delta }_{t+1}|{\Pi }_{<t+1})\stackrel{d}{=}{R}_{t+1}(u,{\epsilon }_{t+1},{\delta }_{t+1}),$$

$$\mathbb{P}\left[{\pi }_{<t,i}\right]=\prod _{t^{\prime }:i^{t^{\prime }}=i}{\mathbb{P}}_{R_{t^{\prime }}}\left[R_{t^{\prime }}(x_i,{\epsilon }_{t^{\prime }},{\delta }_{t^{\prime }})=y_{t^{\prime }}\right].$$

$$\mathbb{P}\left[1\mid x\right]=p_x/2=\frac{{\mathbb{P}}_x\left[{\pi }_{<t,i}\right]}{2{\max }_{x^{\ast }}{\mathbb{P}}_{x^{\ast }}\left[{\pi }_{<t,i}\right]}\in [1/(2e^{\epsilon }),1/2].$$

$$\mathbb{P}\left[0\mid x\right]=(1-p_x/2)\in [1/2,(2e^{\epsilon }-1)/2e^{\epsilon }]$$

$$\mathbb{P}\left[x=x_0\mid \mathsf{accept}=1\right]=$$

$$=$$

$$R(x)\stackrel{d}{=}\gamma \tilde{R}(x)+(1-\gamma )\mu ,$$

$$r(x)=(r(x)-(1-\gamma )r(x_0))+(1-\gamma )r(x_0)$$

$$r(x)-(1-\gamma )r(x_0)=\gamma (r(x_0)+\frac{1}{\gamma }(r(x)-r(x_0)))=\gamma \tilde{r}(x).$$

$$r(x_0)+\frac{1}{\gamma }(r(x)-r(x_0))\ge r(x_0)\left(1+\frac{e^{-{\epsilon }^{\prime }}-1}{\gamma }\right)=r(x_0)e^{-\epsilon }.$$

$${\int }_{\Omega }\tilde{r}(x)={\int }_{\Omega }r(x_0)+\frac{1}{\gamma }{\int }_{\Omega }(r(x)-r(x_0))=1+\frac{1}{\gamma }(1-1)=1,$$

$$\tilde{r}(x)=$$

$$\le$$

$$=$$

$$\le$$

$$\frac{\tilde{r}(x)(o)}{\tilde{r}(x^{\prime })(o)]}\le \frac{e^{\epsilon }r(x_0)(o)}{e^{-\epsilon }r(x_0)(o)}\le e^{2\epsilon }.$$

$$\mathbb{E}[N_i]=\sum _{t=1}^T{\gamma }_{t}2e^{\epsilon }=\frac{2e^{\epsilon }}{1-e^{-\epsilon }}\sum _{t=1}^T(1-e^{-{\epsilon }_t}).$$

$$\mathbb{E}[N_i]\le \frac{2e^{\epsilon }}{1-e^{-\epsilon }}\sum _{t=1}^T{\epsilon }_t\le \left(\frac{2e^{\epsilon }\cdot \epsilon }{1-e^{-\epsilon }}\right)k.$$

$$\mathbb{P}\left[\sum _{i=1}^m{X}_{i,g,r}<m/(4d)\right]\le \exp \left(-\frac{m}{16d}\right)\le 1/(d^4).$$

$$\mathbb{P}\left[Q_r=P_r|Q_1=P_1,...,Q_{r-1}=P_{r-1},W\right]\ge 1-1/d^3,$$

$$\mathbb{E}\left[\sum _{i=1}^m{Y}_{i,g,r}\right]=$$

$$\mathbb{P}\left[Q_r,P_r\text{ have the same }g\text{-th bit}|Q_1=P_1,...,Q_{r-1}=P_{r-1},W\right]$$

$$=$$

$$\ge$$

$$=$$

$$\ge$$

$$\mathbb{P}\left[Q_r=P_r|Q_1=P_1,...,Q_{r-1}=P_{r-1},W\right]\ge 1-u/d^4\ge 1-1/d^3.$$

Full text

We resolve this question. First, we classify locally private protocols by their compositionality, the multiplicative factor $k\geq 1$ by which the sum of a protocol’s single-round privacy parameters exceeds its overall privacy guarantee. We then show how to efficiently transform any fully interactive $k$-compositional protocol into an equivalent sequentially interactive protocol with an $O(k)$ blowup in sample complexity. Next, we show that our reduction is tight by exhibiting a family of problems such that for any $k$, there is a fully interactive $k$-compositional protocol which solves the problem, while no sequentially interactive protocol can solve the problem without at least an $\tilde{\Omega}(k)$ factor more examples.

We then turn our attention to hypothesis testing problems. We show that for a large class of compound hypothesis testing problems — which include all simple hypothesis testing problems as a special case — a simple noninteractive test is optimal among the class of all (possibly fully interactive) tests.

1 Introduction

In the last several years, differential privacy in the local model has seen wide adoption in industry, including at Google [20, 6], Apple [3], and Microsoft [15]. The choice of adopting the local model of differential privacy — in which privacy protections are added at each individual’s device, before data aggregation — instead of the more powerful central model of differential privacy — in which a trusted intermediary is allowed to first aggregate data before adding privacy protections — is driven by practical concerns. Local differential privacy frees the data analyst from many of the responsibilities that come with the stewardship of private data, including liability for security breaches, and the legal responsibility to respond to subpoenas for private data, amongst others. However, the local model of differential privacy comes with its own practical difficulties. The most well known of these is the need to have access to a larger number of users than would be necessary in the central model. Another serious obstacle — the one we study in this paper — is the need for interactivity.

There are two reasons why interactive protocols — which query users adaptively, as a function of the answers to previous queries — pose practical difficulties. The first is that communication with user devices is slow: the communication in noninteractive protocols can be fully parallelized, but for interactive protocols, the number of rounds of interactivity becomes a running-time bottleneck. The second is that user devices can go offline or otherwise become unreachable — and so it may not be possible to return to a previously queried user and pose a new query. The first difficulty motivates the study of noninteractive protocols. The second difficulty motives the study of sequentially interactive protocols [17] — which may pose adaptively chosen queries — but must not pose more than one query to any user (and so in particular never need to return to a previously queried user).

It has been known since [27] that there can be an exponential gap in the sample complexity between noninteractive and interactive protocols in the local model of differential privacy, and that this gap can manifest itself even in natural problems like convex optimization [32, 33]. However, it was not known whether the full power of the local model could be realized with only sequentially interactive protocols. Almost all known lower bound techniques applied only to either noninteractive or sequentially interactive protocols, but there were no known fully interactive protocols that could circumvent lower bounds for sequential interactivity.

1.1 Our Results

We present two kinds of results, relating to the power of sequentially adaptive protocols and non-adaptive protocols respectively. Throughout, we consider protocols operating on datasets that are drawn i.i.d. from some unknown distribution $\mathcal{D}$, and focus on the sample complexity of these protocols: how many users (each corresponding to a sample from $\mathcal{D}$) are needed in order to solve some problem, defined in terms of $\mathcal{D}$.

Sequential Interactivity

We classify locally private protocols in terms of their compositionality. Informally, a protocol is $k$-compositional if the privacy costs $\{\varepsilon^{i}_{j}\}_{j=1}^{r}$ of the local randomizers executed by any user $i$ over the course of the protocol sum to at most $k\varepsilon$, where $\varepsilon$ is the overall privacy cost of the protocol: $\sum_{j}\varepsilon^{i}_{j}\leq k\varepsilon$. When $k=1$, we say that the protocol is compositional. Compositional protocols capture most of the algorithms studied in the published literature, and in particular, any protocol whose privacy guarantee is proven using the composition theorem for $\varepsilon$-differential privacy111Not every protocol is $1$-compositional: exceptions include RAPPOR [20] and the evolving data protocol of Joseph et al. [24]..

Upper Bounds: For any (potentially fully interactive) compositional protocol $M$, we give a generic and efficient reduction that compiles it into a sequentially interactive protocol $M^{\prime}$, with only a constant factor blow-up in privacy guarantees and sample complexity, while preserving (exactly) the distribution on transcripts generated. This in particular implies that up to constant factors, sequentially adaptive compositional protocols are as powerful as fully adaptive compositional protocols. More generally, our reduction compiles an arbitrary $k$-compositional protocol $M$ into a sequentially interactive protocol $M^{\prime}$ with the same transcript distribution, and a blowup in sample complexity of $O(k)$. 2. 2.

Lower Bounds: We show that our upper bound is tight by proving a separation between the power of sequentially and fully interactive protocols in the local model. In particular, we define a family of problems (Multi-Party Pointer Jumping) such that for any $k$, there is a fully interactive $k$-compositional protocol which can solve the problem given sample complexity $n=n(k)$, but such that no sequentially interactive protocol with the same privacy guarantees can solve the problem with sample complexity $\tilde{o}(k\cdot n)$. Thus, the sample complexity blowup of our reduction cannot be improved in general.

Noninteractivity

We then turn our attention to the power of noninteractive protocols. We consider a large class of compound hypothesis testing problems — those such that both the null hypothesis $H_{0}$ and the alternative hypothesis $H_{1}$ are closed under mixtures. For every problem in this class, we show that the optimal locally private hypothesis test is noninteractive. We do this by demonstrating the existence of a simple hypothesis test for such problems. We then prove that this test’s sample complexity is optimal even among the set of all fully interactive tests by extending information theoretical lower bound techniques developed by Braverman et al. [8] and first applied to local privacy by Joseph et al. [25] and Duchi and Rogers [16] to the fully interactive setting.

1.2 Related Work

The local model of differential privacy was introduced by  Dwork et al. [19] and further formalized by Kasiviswanathan et al. [27], who also gave the first separation between noninteractive locally private protocols and interactive locally private protocols. They did so by constructing a problem, Masked Parity, that requires exponentially larger sample complexity without interaction than with interaction. Daniely and Feldman [14] later expanded this result to a different, larger class of problems. Smith et al. [32] proved a similar separation between noninteractive and interactive locally private convex optimization protocols that use neighborhood-based oracles.

Recent work by Acharya et al. [1, 2] gives a qualitatively different separation between the private-coin and public-coin models of noninteractive local privacy. Informally, the public-coin model allows for an additional “half step” of interaction over the private-coin model in the form of coordinated local randomizer choices across users. In this paper, we use the public-coin model of noninteractivity.

Duchi et al. [17] introduced the notion of sequential interactivity for local privacy. They also provided the first general techniques for proving lower bounds for sequentially interactive locally private protocols by bounding the KL-divergence between the output distributions of $\varepsilon$-locally private protocols with different input distributions as a function of $\varepsilon$ and the total variation distance between these input distributions. Bassily and Smith [5] and Bun et al. [7] later generalized this result to $(\varepsilon,\delta)$-locally private protocols, and Duchi et al. [18] obtained an analogue of Assouad’s method for proving lower bounds for sequentially interactive locally private protocols.

More recently, Duchi and Rogers [16] showed how to combine the above analogue of Assouad’s method with techniques from information complexity [22, 8] to prove lower bounds for estimation problems that apply to a restricted class of fully interactive locally private protocols. A corollary of their lower bounds is that several known noninteractive algorithms are optimal minimax estimators within the class they consider. However, their results do not imply any separation between sequential and full interaction. Moreover, our reduction implies that every (arbitrarily interactive) compositional locally private algorithm can be reduced to a sequentially interactive protocol with only constant blowup in sample complexity, and as a result all known lower bounds for sequentially interactive protocols also hold for arbitrary compositional protocols.

Canonne et al. [10] study simple hypothesis testing under the centralized model of differential privacy, and Theorem 1 of Duchi et al. [17] implies a tight lower bound for sequentially interactive locally private simple hypothesis testing. We extend this lower bound to the fully interactive setting and match it with a noninteractive upper bound for a more general class of compound testing problems that includes simple hypothesis testing as a special case.

Finally, recent subsequent work [26] gives a stronger exponential sample complexity separation separation between the sequentially and fully interactive models. It does so through a general connection between communication complexity and sequentially interactive sample complexity. Applying this connection to a communication problem similar to the “multi-party pointer jumping” described in Section 4 completes the result.

2 Preliminaries

We begin with the definition of approximate differential privacy. Given data domain $\mathcal{X}$, two data sets $S,S^{\prime}\in\mathcal{X}^{n}$ are neighbors (denoted $S\sim S^{\prime}$) if they differ in at most one coordinate: i.e. if there exists an index $i$ such that for all $j\neq i$, $S_{j}=S^{\prime}_{j}$. A differentially private algorithm must have similar output distributions on all pairs of neighboring datasets.

Definition 2.1** ([19]).**

Let $\varepsilon,\delta\geq 0$. A randomized algorithm $\mathcal{M}:\mathcal{X}^{n}\rightarrow\mathcal{O}$ is $(\varepsilon,\delta)$-differentially private if for every pair of neighboring data sets $S\sim S^{\prime}\in\mathcal{X}^{n}$, and every event $\Omega\subseteq\mathcal{O}$

[TABLE]

When $\delta=0$, we say that $\mathcal{M}$ satisfies (pure) $\varepsilon$-differential privacy.

Differential privacy has two nice properties. First, it composes neatly: the composition of algorithms $\mathcal{M}_{1},\ldots,\mathcal{M}_{n}$ that are respectively $(\varepsilon_{1},\delta_{1}),\ldots,(\varepsilon_{n},\delta_{n})$-differentially private is $(\sum_{i}\varepsilon_{i},\sum_{i}\delta_{i})$-differentially private. For pure differential privacy, this is tight in general. Second, differential privacy is resilient to post-processing: given an $(\varepsilon,\delta)$-differentially private $\mathcal{M}$ and any function $f$, $f\circ\mathcal{M}$ is still $(\varepsilon,\delta)$-differentially private (see Appendix A.1 for details). For brevity, we often abbreviate “differential privacy” as “privacy”.

As defined, the constraint of differential privacy is on the output of an algorithm $\mathcal{M}$, not on its internal workings. Hence, it implicitly assumes a trusted data curator, who has access to the entire raw dataset. This is sometimes referred to as differential privacy in the central model. In contrast, this paper focuses on the more restrictive local model [19] of differential privacy. In the local model, the private computation is an interaction between $n$ users, each of whom hold exactly one dataset record, and is coordinated by a protocol $\mathcal{A}$. We assume throughout this paper that each user’s datum is drawn i.i.d. from some unknown distribution: $x_{i}\sim_{iid}\mathcal{D}$222Roughly speaking, this corresponds to a setting in which users are “symmetric” and in which nothing differentiates them a priori. All of our results generalize to the setting in which there are different “types” of users, known to the protocol up front.. Informally, at each round $t$ of the interaction, a protocol $\mathcal{A}$ observes the transcript of interactions so far, selects a user, and assigns the user a randomizer. The user then applies the randomizer to their datum, using fresh randomness for each application, and publishes the output. In turn, the protocol observes the updated transcript, selects a new user-randomizer pair, and the process continues. We define these terms precisely below.

Definition 2.2**.**

An $(\varepsilon,\delta)$-randomizer $R\colon X\to Y$ is an $(\varepsilon,\delta)$-differentially private function taking a single data point as input.

A simple, canonical, and useful randomizer is randomized response [34, 19].

Example 2.1** (Randomized Response).**

Given data universe $\mathcal{X}=[k]$ and datum $x_{i}\in\mathcal{X}$, $\varepsilon$-randomizer $\mathsf{RR}\left(x_{i},\varepsilon\right)$ outputs $x_{i}$ with probability $\tfrac{e^{\varepsilon}}{e^{\varepsilon}+k-1}$ and otherwise outputs a uniformly random element of $\mathcal{X}-\{x_{i}\}$.

Next, we formally define transcripts and protocols.

Definition 2.3**.**

A transcript $\pi$ is a vector consisting of 5-tuples $(i^{t},R_{t},\varepsilon_{t},\delta_{t},y_{t})$ — encoding the user chosen, randomizer assigned, randomizer privacy parameters, and randomized output produced — for each round $t$. $\pi_{<t}$ denotes the transcript prefix before round $t$. Letting $S_{\pi}$ denote the collection of all transcripts and $S_{R}$ the collection of all randomizers, a protocol is a function $\mathcal{A}\colon S_{\pi}\to\left([n]\times S_{R}\times\mathbb{R}_{\geq 0}\times\mathbb{R}_{\geq 0}\right)\cup\{\perp\}$ mapping transcripts to users, randomizers, and randomizer privacy parameters ($\perp$ is a special character indicating a protocol halt).

The transcript that results from running a locally private computation will often be post-processed to compute some useful function of the data. However, the privacy guarantee must hold even if the entire transcript is observed. Hence, in this paper we abstract away the task that the computation is intended to solve, and view the output of a locally private computation as simply the transcript it generates.

To clarify the role of interaction in these private computations – especially when analyzing reductions between computations with different kinds of interactivity – it is often useful to speak separately of protocols and experiments. While the protocol $\mathcal{A}$ is a function mapping transcripts to users and randomizers, the experiment is the interactive process that maps a protocol and collection of users drawn from a distribution $\mathcal{D}$ to a finished transcript. In the simplest case, $\mathsf{FollowExpt}$ (Algorithm 1), the experiment exactly follows the outputs of its protocol.

However, experiments may in general heed, modify, or ignore the outputs of their input protocol. We delineate the privacy characteristics of experiment-protocol pairs and protocols in isolation below. Here and throughout, the dataset is not viewed as an input to an experiment, but is drawn from $\mathcal{D}$ by the experiment-protocol pair. Drawing a fresh user $\sim\mathcal{D}$ corresponds to adding an additional data point, and so the sample complexity of an experiment-protocol pair is the number of draws from $\mathcal{D}$ over the run of the algorithm. For the simple algorithm $\mathsf{FollowExpt}(\mathcal{A})$ defined above, the sample complexity is always $n$. Finally we remark that although the distribution $\mathcal{D}$ and the sample complexity $n$ are inputs to the experiment, for brevity we typically omit them and focus on the protocol $\mathcal{A}$; e.g. writing $\mathsf{Expt}(\mathcal{A})$ rather than $\mathsf{Expt}(\mathcal{A},\mathcal{D},n)$.

Definition 2.4**.**

Experiment-protocol pair $\mathsf{Expt}(\mathcal{A})$ satisfies $(\varepsilon,\delta)$-local differential privacy (LDP) if it is $(\varepsilon,\delta)$-differentially private in its transcript outputs. A protocol $\mathcal{A}$ satisfies $(\varepsilon,\delta)$-local differential privacy (LDP) if experiment-protocol pair $\mathsf{FollowExpt}(\mathcal{A})$ is $(\varepsilon,\delta)$-locally differentially private.

Experiment-protocol pairs can be, by increasing order of generality, noninteractive, sequentially interactive, and fully interactive.

Definition 2.5**.**

An experiment-protocol pair $\mathsf{Expt}(\mathcal{A})$ is noninteractive if, at each round $t$, as random variables, $(i^{t},R_{t},\varepsilon_{t},\delta_{t})\perp\!\!\!\!\perp\Pi_{<t}\mid t$.

In other words, noninteractivity forces nonadaptivity, and all user-randomizer assignments are made before the experiment begins. In contrast, in sequentially interactive experiment-protocol pairs, users may be queried adaptively, but only once.

Definition 2.6**.**

An experiment-protocol pair $\mathsf{Expt}(\mathcal{A})$ is sequentially interactive if, at each round $t$, $i^{t}\neq i^{t-1},\ldots,i^{1}$.

Finally, in in fully interactive experiments, the experiment-protocol may make user-randomizer assignments adaptively, and each user may receive arbitrarily many randomizer assignments. Along the same lines, we say a protocol $\mathcal{A}$ is noninteractive (respectively sequentially and fully interactive) if $\mathsf{FollowExpt}(\mathcal{A})$ is a noninteractive (respectively sequentially and fully interactive) experiment-protocol pair. This experiment-protocol formalism will be useful in constructing the full-to-sequential reduction in Section 3; elsewhere, we typically elide the distinction and simply reason about $\mathsf{FollowExpt}(\mathcal{A})$ as “protocol $\mathcal{A}$”. For any locally private protocol, we refer to the number of users $n$ that it queries as its sample complexity. For fully interactive protocols, the total number of rounds — which we denote by $T$ — may greatly exceed $n$. In contrast, for both non-interactive and sequentially interactive protocols, the number of rounds $T\leq n$.

At each round $t$ of a fully interactive $\varepsilon$-locally private protocol, we know that $\varepsilon_{t}\leq\varepsilon$. For many protocols, we can say more about how the $\varepsilon_{t}$ parameters relate to $\varepsilon$:

Definition 2.7**.**

Consider an $\varepsilon$-locally private protocol $\mathcal{A}$. Let $\{\varepsilon_{t}\}_{t=1}^{T}$ denote the minimal privacy parameters of the local randomizers $R_{t}$ selected at round $t$ considered as random variables. We say the protocol $\mathcal{A}$ is $k$-compositionally private if for all $i\in[n]$, with probability $1$ over the randomness of the transcript,

[TABLE]

If $k=1$, a protocol is simply compositional private.

Remark**.**

In fact, all of our results hold without modification even under the weaker condition of average $k$-compositionality. For a protocol $\mathcal{A}$ with sample complexity $n$, $\mathcal{A}$ is $k$-compositional on average if

[TABLE]

For brevity, we often shorthand “$k$-compositionally private” as simply “$k$-compositional”.

Informally, a compositionally private protocol is one in which the privacy parameters for each user “just add up.” Almost every locally private protocol studied in the literature (and in particular, every protocol whose privacy analysis follows from the composition theorem for pure differential privacy) is compositionally private333This simple compositionality applies even if $\{\varepsilon_{t}\}_{t=1}^{T}$ are chosen adaptively in each round (see Theorem 3.6 in Rogers et al. [30]).. They are so ubiquitous that it is tempting to guess that all $(\varepsilon,0)$-locally private protocols are compositional. However, this is false: for every $k$ and $\varepsilon$, there are $\varepsilon$-locally private protocols that fail to be $k$-compositionally private. The following example shows that by taking advantage of special structure in the data domain and choice of randomizers it is possible to achieve $(\varepsilon,0)$-local privacy, even as the sum of the round-by-round privacy parameters greatly exceeds $\varepsilon$.

Example 2.2** (Informal).**

Let the data universe $\mathcal{X}$ consist of the canonical basis vectors $e_{1},\ldots,e_{d}\in\{0,1\}^{d}$, and let each $x_{1},\ldots,x_{n}$ be an arbitrary element of $\mathcal{X}$. Consider the $d$ round protocol where, for each round $j\in[d]$, every user $i$ with $x_{i}=e_{j}$ outputs a sample from $\mathsf{RR}\left(1,\varepsilon\right)$, and the remaining users output a sample from $\mathsf{Ber}\left(0.5\right)$. As $\mathsf{RR}(\cdot,\varepsilon)$ is an $\varepsilon$-local randomizer which each user employs only once, and remaining outputs are data-independent, this protocol is $\varepsilon$-locally private. But the protocol fails to be $k$-compositionally private for $k<d/2$.

The preceding example demonstrates that the careful choice of local randomizers based on the data universe structure can strongly violate compositional privacy. Seen another way, when multiple queries are asked of the same user, there are situations in which the correlation in privatized responses induced by being run on the same data element can lead to arbitrarily sub-compositional privacy costs. The main result of our paper is that the additional power of a fully interactive protocol, on top of sequential interactivity, is characterized by its compositionality.

3 From Full to Sequential Interactivity

We show that any $(\varepsilon,0)$-locally private compositional protocol is “equivalent” to a sequentially interactive protocol with sample complexity that is larger by only a small constant factor. By equivalent, we mean that for any $(\varepsilon,0)$-locally private compositional protocol, we can exhibit a sequentially interactive $(3\varepsilon,0)$-locally differentially private protocol with only a constant factor larger sample complexity that induces exactly the same distribution on transcripts. Thus for any task for which the original protocol was useful, the sequentially interactive protocol is just as useful444Formally, for any loss function defined over a data distribution $\mathcal{D}$ and a transcript $\Pi$, when data points $x_{i}$ are drawn i.i.d. from $\mathcal{D}$, the two protocols induce exactly the same distribution over transcripts, and hence the same distribution over losses. Once one restricts attention to locally private protocols with privacy parameter $\varepsilon\leq 1$ that take as input points drawn i.i.d. from a distribution $\mathcal{D}$, it is without loss of generality to measure the success or failure of a protocol with respect to the underlying distribution $\mathcal{D}$, rather than with respect to the sample. This is because such protocols are $\approx\varepsilon/\sqrt{n}$ differentially private when viewed in the central model of differential privacy (in which the input may be permuted before used in the protocol) [21, 4], and hence the distribution on transcripts would be almost unchanged even if the entire dataset was resampled i.i.d. from $\mathcal{D}$. [13, 28]. Thus, for such protocols, the transcript distribution is governed by the data distribution $\mathcal{D}$, but not (significantly) by the sample..

More generally, we give a generic reduction under which any $(\varepsilon,0)$-private $k$-compositional protocol can be compiled into a sequentially interactive protocol with an $e^{\varepsilon}k$-factor increase in sample complexity.

Our proof is constructive; given an arbitrary $k$-compositional $(\varepsilon,0)$-locally differentially private protocol we show how to simulate it using a sequentially interactive protocol that induces the same joint distribution on transcripts. The “simulation” is driven by three main ideas:

Bayesian Resampling: The dataset used in a locally differentially private protocol is static once the protocol begins. However, we consider the following thought experiment: each user’s datum is resampled from the posterior distribution on their datum, conditioned on the transcript thus far, before every round in which they are given a local randomizer. We observe that the mechanism from this thought experiment induces exactly the same joint distribution on datasets and transcripts upon completion of the mechanism. Thus, for the remainder of the argument, we can seek to simulate this “Bayesian Resampling” version of the mechanism. 2. 2.

Private Rejection Sampling: Because of the local differential privacy guarantee, at any step of the algorithm, the posterior on a user’s datum conditioned on the private transcript generated so far must be close to their prior. Thus, it is possible to sample from this posterior distribution by first sampling from the prior, and then applying a rejection sampling step that is both a) likely to succeed, and b) differentially private. Sampling from the prior simply corresponds to querying a new user. At first glance, applying rejection sampling as needed seems to require information that the users will not have available, because they do not know the underlying data distribution $\mathcal{D}$. But an application of Bayes rule, together with a data independent rescaling can be used to re-write the required rejection probability using only quantities that each user can compute from her own data point and the transcript. A similar use of rejection sampling appears in the simulation of locally private algorithms by statistical query algorithms given by Kasiviswanathan et al. [27]. 3. 3.

Data Independent Decomposition of Local Randomizers: The two ideas above suffice to transform a fully interactive mechanism into a sequentially interactive mechanism, with a blowup in sample complexity from $n$ to $T$ (because in the sequentially interactive protocol that results from rejection sampling, each user applies only one local randomizer instead of an average of $T/n$). However, we generalize a recent result of [4] to show that any $\varepsilon_{i}$-private local randomizer can be described as a mixture between a data independent distribution and an $(\varepsilon,0)$-private local randomizer for any $\varepsilon>\varepsilon_{i}$, where the weight on the data independent distribution is roughly (for small constant $\varepsilon$) $1-\varepsilon_{i}/\varepsilon$. Thus we can simulate each local randomizer while only needing to query a new user with probability $\varepsilon_{i}/\varepsilon$. As a result, for any compositional mechanism, 1 user in the sequential setting suffices (in expectation) to simulate the entire transcript of a single user in the fully interactive setting. More generally, if the mechanism is $k$-compositional, then $k$ users are required in expectation to carry out the simulation. The realized sample complexity concentrates sharply around its expectation.

3.1 Step 1: A Bayesian Thought Experiment

The first step of our construction is to observe that for any locally private protocol $\mathcal{A}$, $\mathsf{BayesExpt}(\mathcal{A})$ induces exactly the same distribution over transcripts as $\mathsf{FollowExpt}(\mathcal{A})$. The difference is that in $\mathsf{BayesExpt}(\mathcal{A})$, between each interaction with a given user $i$, their datum $x_{i}$ is resampled from the posterior distribution on user $i$’s data conditioned on the portion of the transcript generated thus far. We prove in Lemma 3.1 that the two experiments produce exactly the same transcript distribution. Once we establish this, our goal will be to simulate the transcript distribution induced by $\mathsf{BayesExpt}(\mathcal{A})$.

Note that when $i^{t}$ is selected for the first time $Q_{i,t}=\mathcal{D}$, and so the sample complexity (e.g. number of draws from $\mathcal{D}$) of $\mathsf{BayesExpt}(\mathcal{A})$ is bounded by $n$.

Lemma 3.1**.**

For any protocol $\mathcal{A}$, Let $\Pi^{f}$ be the transcript random variable that is output by $\mathsf{FollowExpt}(\mathcal{A})$ and let $\Pi^{b}$ be the transcript output by $\mathsf{BayesExpt}(\mathcal{A})$. Then

[TABLE]

where $\stackrel{{\scriptstyle d}}{{=}}$ denotes equality of distributions.

Proof.

We show this by (strong) induction on rounds in the transcript. The base case $t=1$ is immediate: for any index $i^{1}$ selected by $\mathsf{BayesExpt}(\mathcal{A})$, the posterior distribution $Q_{i,1}$ is the same as the prior $\mathcal{D}$.

Now suppose it is true up to time $t+1$, i.e. $\Pi^{f}_{<t+1}\stackrel{{\scriptstyle d}}{{=}}\Pi^{b}_{<t+1}$. Then since the joint distributions $\Pi_{<t+2}$ factor as $(i^{t+1},R_{t+1},\varepsilon_{t+1},\delta_{t+1},Y_{t+1}|\Pi_{<t+1})\cdot\Pi_{<t+1}$, it suffices to show that the conditional distributions on $i^{t+1},R_{t+1},\varepsilon_{t+1},\delta_{t+1},Y_{t+1}|\Pi_{<t+1}$ coincide. Moreover, the conditional distribution on $i^{t+1},R_{t+1},\varepsilon_{t+1},\delta_{t+1}|\Pi_{<t+1}$ is given by $\mathcal{A}(\Pi_{<{t+1}})$ under both algorithms, and so it remains only to show that $Y_{t+1}|i^{t+1},R_{t+1},\varepsilon_{t+1},\delta_{t+1},\Pi_{<t+1}$ is the same distribution under both algorithms. Under $\mathsf{FollowExpt}(\mathcal{A})$,

[TABLE]

where $u\stackrel{{\scriptstyle d}}{{=}}x_{i^{t+1}}|\Pi_{<t+1}\stackrel{{\scriptstyle d}}{{=}}Q_{i,t+1}$ by definition, and we use the fact that after conditioning on $\Pi_{<t+1}$, $x_{i^{t+1}}$ is independent of $\varepsilon_{t+1}$ and $\delta_{t+1}$. Redrawing $u\sim Q_{i,t+1}$ does not change the marginal distribution of $R_{t+1}(u,\varepsilon_{t+1},\delta_{t+1})$, which is exactly the distribution under $\mathsf{BayesExpt}(\mathcal{A})$, as desired. ∎

3.2 Step 2: Sequential Simulation of Algorithm 2 via Rejection Sampling

We now show how to replace step $8$ in Algorithm 2 by selecting a new datapoint (drawn from $\mathcal{D}$) at every round and using rejection sampling to simulate a draw from $Q_{i,t}$. The result is a sequentially interactive mechanism that preserves the transcript distribution of Algorithm 2 (and, by Lemma 3.1, of Algorithm 1), albeit one with a potentially very large increase in sample complexity (from $n$ to $T$). The rejection sampling step increases the privacy cost of the protocol by at most a factor of $2$.

We first review why it is non-obvious that rejection sampling can be performed in this setting. We want to sample from the target distribution $Q_{i,t}$, the posterior $x_{i}^{t}|\pi_{<t}$, using samples from the proposal distribution $\mathcal{D}$. Let $p_{\pi}$ denote the density function of $Q_{i,t}$ and let $p$ denote the density function of $\mathcal{D}$. In rejection sampling, we would typically sample $u\sim\mathcal{D}$, and with probability $\propto\frac{p_{\pi}(u)}{p(u)}$ we would accept $u$ as a sample drawn from $Q_{i,t}$, or else redraw another $u$ and continue.

This is not immediately possible in our setting, since the individuals (who must perform the rejection sampling computation) do not know the prior density $p$ and hence do not know the posterior $p_{\pi}$. As a result, they cannot compute either the numerator or denominator of the expression for the acceptance probability. We solve this problem by using the fact that we are simulating a posterior with a prior distribution, and formulate the rejection sampling probability ratio as a quantity depending only on a user’s private data point and the transcript. Users may then compute this quantity themselves.

To define our transformed rejection sampler we set up some new notation: given a user $i$ and round $t$, let $\pi_{<t,i}$ denote the subset of the realized transcript up to time $t$ that corresponds to user $i$’s data, i.e. $\pi_{<t,i}=\{(i^{t^{\prime}},R_{t^{\prime}},\varepsilon_{t^{\prime}},\delta_{t^{\prime}},y_{t^{\prime}}):t^{\prime}<t,i^{t^{\prime}}=i\}$. Let $\mathbb{P}_{x_{i}}\left[\pi_{<t,i}\right]$ denote the conditional probability of the messages corresponding to user $i$ given the choices of privacy parameters and randomizers up to time $t$:

[TABLE]

Using this notation, we define our rejection sampling procedure $\mathsf{RejSamp}$ in Algorithm 3.

We now prove that $\mathsf{RejSamp}$ is private and does not need to sample many users.

Lemma 3.2**.**

Let $Y_{t}\stackrel{{\scriptstyle d}}{{=}}R_{t}(x^{\prime}),$ where $x^{\prime}\sim Q_{i,t}$ and let $Y_{t}^{\prime}$ be defined by the rejection sampling algorithm $\mathsf{RejSamp}$ above. Let the sample complexity $N$ be the total number of new users $x$ drawn in step 4 of $\mathsf{RejSamp}$. Then $\mathsf{RejSamp}$ is $(\varepsilon+\varepsilon_{t},0)$-locally private, $Y_{t}\stackrel{{\scriptstyle d}}{{=}}Y_{t}^{\prime}$, and $\mathbb{E}[N]\leq 2e^{\varepsilon}$.

Proof of Lemma 3.2.

Claim 3.3**.**

$\mathsf{RejSamp}$* is $(\varepsilon+\varepsilon_{t})$-locally private.*

We first show that publishing a draw from $\mathsf{Ber}\left(p_{x}/2\right)$ is $(\varepsilon,0)$-locally private. By assumption publishing $\pi_{<t}$, and hence publishing $\pi_{<t,i}$ (by post-processing), $(\varepsilon,0)$-private. Hence for any $x\in\mathcal{X}$

[TABLE]

Therefore for any $x,x^{\prime}$, $\mathbb{P}\left[1\mid x\right]\leq e^{\varepsilon}\mathbb{P}\left[1\mid x^{\prime}\right]$. Similarly,

[TABLE]

and by $1+x\leq e^{x}$, we get $1-\varepsilon\leq e^{-\varepsilon}$, so $1+\varepsilon\geq 2-e^{-\varepsilon}$, and $e^{\varepsilon}/2\geq(2^{\varepsilon}-1)/(2e^{\varepsilon})$. Thus for any $x,x^{\prime}$, $\mathbb{P}\left[0\mid x\right]\leq e^{\varepsilon}\mathbb{P}\left[0\mid x^{\prime}\right]$.

Releasing $R_{t}(x,\varepsilon_{t})$ is $\varepsilon_{t}$-locally private, so by composition the whole process is $(\varepsilon+\varepsilon_{t})$-locally private.

Claim 3.4**.**

$Y_{t}\stackrel{{\scriptstyle d}}{{=}}Y_{t}^{\prime}$**

It suffices to show that $x|\{\mathsf{accept}=1\}\stackrel{{\scriptstyle d}}{{=}}Q_{i,t}$. Fix any $x_{0}\in\mathcal{X}$. Then by Bayes’ rule

[TABLE]

as desired. Finally, since $p_{x}/2\geq\frac{1}{2e^{\varepsilon}}$, the expected number of samples until $\mathsf{accept}=1$ is $\leq 2e^{\varepsilon}$. ∎

3.3 Step 3: Data Independent Decomposition of Local Randomizers

The preceding sections enable us to simulate a fully interactive $k$-compositional $(\varepsilon,0)$-locally private protocol with a sequentially interactive $(2\varepsilon,0)$-locally private protocol. However, our solution so far may require sampling a new user for each query in the original protocol. Since a fully interactive protocol’s query complexity may greatly exceed its sample complexity, this is undesirable. To address this problem, we decompose each local randomizer in a way that substantially reduces the number of queries that actually require samples.

Let $R:\mathcal{X}\to\mathcal{Y}$ be an $\varepsilon^{\prime}$ local randomizer, fix an arbitrary element $x_{0}\in\mathcal{X}$, and let $x$ be a private input to $R$. Then Lemma 5.2 in Balle et al. [4] shows that we can write $R(x)$ as a mixture $\gamma w+(1-\gamma)d_{x}$, where $w$ is a data-independent distribution, $d_{x}$ is a data-dependent distribution, and $\gamma\geq e^{-\varepsilon^{\prime}}$. This suggests that decomposition — by answering a proportion of queries from data-independent distributions — can reduce the sample complexity of our solution. Unfortunately, the data dependent distribution need not be differentially private (in fact, it often corresponds to a point mass on the private data point), so the privacy of the overall mechanism crucially relies on not releasing which of the two mixture distributions the output was sampled from.

We first generalize this result, showing that for any $\varepsilon\geq\varepsilon^{\prime}$, we can write $R(x)$ as $(1-\gamma)w+\gamma\tilde{R}(x)$ where $\tilde{R}$ is a $2\varepsilon$-differentially private local randomizer, and $\gamma=\frac{e^{-\varepsilon^{\prime}}-1}{e^{-\varepsilon}-1}$ (Lemma 3.5). The upshot of this generalization is that even if we make public which part of the mixture distribution was used, the resulting privacy loss is still bounded by $2\varepsilon$. Larger values of $\varepsilon$ increase our chance of sampling from a data-independent distribution when simulating a local randomizer, while increasing the privacy cost incurred by a user in the event that we sample from the data-dependent mixture component. This tradeoff will be crucial for us in the proof of our main result.

Lemma 3.5** (Data Independent Decomposition).**

Let $R:\mathcal{X}\to\mathcal{Y}$ be an $\varepsilon^{\prime}$-differentially private local randomizer and let $\varepsilon\geq\varepsilon^{\prime}$. Then there exists a mapping $\tilde{R}$ and fixed data-independent distribution $\mu$ such that $\tilde{R}(\cdot)$ is a $2\varepsilon-$differentially private local randomizer and

[TABLE]

where $\gamma=\frac{e^{-\varepsilon^{\prime}}-1}{e^{-\varepsilon}-1}$.

Proof.

Let $0<\varepsilon^{\prime}\leq\varepsilon$, fix any $x_{0}\in\mathcal{X}$, let $\gamma=\frac{e^{-\varepsilon^{\prime}}-1}{e^{-\varepsilon}-1}$, and let $r(x)$ denote the density function of the local randomizer $R$ with input $x$ implicitly evaluated at some arbitrary point in the range, which we suppress. Since $\varepsilon\geq\varepsilon^{\prime}>0$, $\gamma\in(0,1]$ is a valid mixture probability. Thus we can write

[TABLE]

and, rewriting the first term,

[TABLE]

$\tilde{r}$ defines a new mapping $\tilde{R}(\cdot)$ by mapping $x$ to the random variable $\tilde{R}(x)$ with density function $\tilde{r}(x)=(r(x_{0})+\frac{1}{\gamma}(r(x)-r(x_{0})))$. Thus, it suffices to show that the mapping $\tilde{R}(x)$ is a $2\varepsilon$-private local randomizer.

We first show that for any $x$, $\tilde{r}(x)$ is a well-defined density function. Since $R$ is an $\varepsilon^{\prime}$-private local randomizer, $r(x)-r(x_{0})\geq(e^{-\varepsilon^{\prime}}-1)r(x_{0})$, and so

[TABLE]

This establishes that $\tilde{r}(x)$ is non-negative. Then since

[TABLE]

$\tilde{r}(x)$ defines a valid density function for any $x$.

To see that $\tilde{r}$ is also a $2\varepsilon$-private local randomizer, fix any outcome $o\in\mathcal{Y}$ and any other $x^{\prime}\in\mathcal{X}$. Since $r$ is an $\varepsilon^{\prime}$-local randomizer, $r(x)-r(x_{0})\leq r(x_{0})(e^{\varepsilon^{\prime}}-1)$ and we get

[TABLE]

We already showed $\tilde{r}(x^{\prime})\geq e^{-\varepsilon}r(x_{0})$, so

[TABLE]

∎

3.4 Putting it All Together: The Complete Simulation

Finally, we combine rejection sampling and decomposition to give our complete reduction, Algorithm 4. We use rejection sampling to convert from a fully interactive mechanism to a sequentially interactive one and use our data-independent decomposition of local randomizers to reduce the sample complexity of the converted mechanism.

We now prove that $\mathsf{Reduction}$ has the desired interactivity, privacy, transcript, and sample complexity guarantees. We again denote by $N$ the sample complexity of $\mathsf{Reduction}$, i.e. the number of samples drawn from the prior $\mathcal{D}$ over the run of the algorithm, either in Step $15$ (which is bounded by $n$), or over the runs of $\mathsf{RejSamp}$ in line $10$. We observe that sampling from the prior $\mathcal{D}$ simply corresponds to using a new datapoint drawn from $\mathcal{D}$. Fixing a protocol $\mathcal{A}$, let $\Pi^{r}$ denote the transcript random variable generated by $\mathsf{Reduction}(\mathcal{A})$, and let $\Pi^{b}$ denote the transcript random variable generated by $\mathsf{BayesExpt}(\mathcal{A})$.

Theorem 3.6**.**

Let $\mathcal{A}$ a fully-interactive $k$-compositional $(\varepsilon,0)$-locally private protocol. Then

$\mathsf{Reduction}(\mathcal{A})$* is sequentially interactive,* 2. 2.

$\mathsf{Reduction}(\mathcal{A})$* is $(3\varepsilon,0)$-locally private,* 3. 3.

$\Pi^{r}\stackrel{{\scriptstyle d}}{{=}}\Pi^{b}$, 4. 4.

$\mathbb{E}[N]\leq n(\frac{2e^{\varepsilon}\cdot\varepsilon}{1-e^{-\varepsilon}}k+1)$, and with probability $1-\delta$, $N=O(nk+\sqrt{nk\log\frac{1}{\delta}})$.

Proof of Theorem 3.6.

1. Interactivity: Since each user $i$’s data is only used once (before $s_{i}$ is set to $1$), $\mathsf{Reduction}$ is sequentially interactive.

2. Privacy: Consider a data point $x$ corresponding to an arbitrary user over the run of $\mathsf{Reduction}(\mathcal{A})$. Then either $x$ is drawn in line $18$, or $x$ is drawn during a rejection sampling step. In the first case, $x$ is only used once in step $18$, as input to an $\varepsilon_{t}$-local randomizer, preserving $(\varepsilon,0)$-LDP, since $\varepsilon_{t}\leq\varepsilon$. If $x$ is drawn during the rejection sampling step, then it is used during the use of rejection sampling to simulate a draw from a $(2\varepsilon,0)$-local randomizer $\tilde{R}(\cdot)$, where the input transcript $\pi_{<t}$ has been generated $(\varepsilon,0)$-privately. The privacy of the input transcript is relevant because it bounds the privacy of the user’s rejection sampling step. By Lemma 3.2, this is $(3\varepsilon,0)$-private.

3. Transcripts: We prove this claim by a similar argument as that of Lemma 3.1: we show by induction that the transcript distribution at each step $t$ is the same for $\mathsf{Reduction}(\mathcal{A})$ and $\mathsf{BayesExpt}(\mathcal{A})$. This is trivially true at $t=1$. Now suppose it is true up to time $t+1$, i.e. $\Pi^{r}_{<t+1}\stackrel{{\scriptstyle d}}{{=}}\Pi^{b}_{<t+1}$. Then since the joint distributions $\Pi_{<t+2}$ factor as $(i^{t+1},R_{t+1},\varepsilon_{t+1},Y_{t+1}|\Pi_{<t+1})\cdot\Pi_{<t+1}$, it suffices to show that the conditional distributions on $i^{t+1},R_{t+1},\varepsilon_{t+1},Y_{t+1}|\Pi_{<t+1}$ coincide.

Note that under both $\mathsf{Reduction}(\mathcal{A})$ and $\mathsf{BayesExpt}(\mathcal{A})$, protocol $\mathcal{A}$ is used to select $i^{t+1},R_{t+1},\varepsilon_{t+1}$ as a function of $\Pi_{<t+1}$, so we can condition on $i^{t+1},R_{t+1},\varepsilon_{t+1}$ as well, and need only show that the distribution on $Y_{t+1}$ is the same. Under $\mathsf{BayesExpt}(\mathcal{A})$, $Y_{t+1}$ is drawn from $R_{t+1}(u,\varepsilon_{t+1}),u\sim Q_{i,t+1}$. There are two cases for $\mathsf{Reduction}(\mathcal{A})$:

• •

If $s_{i}^{t+1}=0$, then under $\mathsf{Reduction}(\mathcal{A}),Y_{t+1}$ is drawn in line $18$ from $R_{t+1}(u,\varepsilon_{t+1}),u\sim Q_{i,t+1}$, as desired.

• •

If $s_{i}^{t+1}=1$, then $\mathsf{Reduction}(\mathcal{A})$ uses Lemma 3.5 to write $R_{t+1}(\cdot)$ as a mixture. Hence if we sample from the mixture with input $u\sim Q_{i,t+1}$, we sample from $R_{t+1}(u)$, which is the desired sampling distribution. To see that $\mathsf{Reduction}(\mathcal{A})$ does sample from the target, we need only show that $Y_{t+1}$ drawn in line $13$ is sampled from $\tilde{R}_{t}(u)$ where $u\sim Q_{i,t+1}$. This is true by Lemma 3.2.

4. Sample Complexity: Here we bound the expected sample complexity, deferring the high probability bound to Section A.3 in the Appendix. Let $N_{i}$ be the number of fresh samples drawn over all rounds $t$ where $i^{t}=i$, i.e. the number of samples drawn when simulating follow-up queries to $i$. Let $N_{i}^{t}$ be the number of samples drawn during rejection sampling in round $t$; we imagine that regardless of the coin-flip in line $11$ of the pseudocode of $\mathsf{Reduction}$, $N_{i}^{t}$ is always drawn. Then the total number of samples is $N_{i}=\sum_{t=1}^{T}\gamma_{t}N_{i}^{t}$. (Note that for simplicity, we are summing over all rounds $T$, since equivalently we may imagine that each user is given a local randomizer at each round, with privacy cost [math] in any round in which $i_{t}\neq i$.) Then by Lemma 3.2

[TABLE]

Since $1-x\leq e^{-x}$, we get that $1-\varepsilon_{t}\leq e^{-\varepsilon_{t}}$ and so $1-e^{-\varepsilon_{t}}\leq\varepsilon_{t}$. Hence

[TABLE]

Summing over $i$, and including the at most $n$ samples drawn in line $18$ bounds the expected sample complexity by $((\frac{2e^{\varepsilon}\cdot\varepsilon}{1-e^{-\varepsilon}})k+1)n$, as desired. ∎

4 Separating Full and Sequential Interactivity

We now prove that our reduction in Section 3 is tight in the sense that any generic reduction from a fully interactive protocol to a sequentially interactive protocol must have a sample complexity blowup of $\tilde{\Omega}(k)$ when applied to a $k$-compositional protocol. Specifically, we define a family of problems such that for every $k$, there is a fully interactive $k$-compositional protocol that can solve the problem with sample complexity $n=n(k)$, but such that any sequentially interactive protocol solving the problem must have sample complexity $\tilde{\Omega}(k\cdot n)$.

Informally, the family of problems (Multi-Party Pointer Jumping, or $\mathcal{MPJ}(d)$) we introduce is defined as follows. An instance of $\mathcal{MPJ}(d)$ is given by a complete tree of depth $d$. Every vertex of the tree is labelled by one of its children. By following the labels down the tree, starting at the root, an instance defines a unique root-to-leaf path. Given an instance of $\mathcal{MPJ}(d)$, the data distribution is defined as follows: to sample a new user, first select a level of the tree uniformly $\ell\in[d]$ at random, and provide that user with the vertex-labels corresponding to level $\ell$ (note that fixing an instance of the problem, every user corresponding to the same level of the tree has the same data). The problem we wish to solve privately is to identify the unique root to leaf path specified by the instance.

We first show that there is a fully interactive protocol which can solve this problem with sample complexity $n=\tilde{O}(d^{2}/\varepsilon^{2})$. The protocol is $k$-compositional for $k=\Theta(d)$. Roughly speaking, the protocol works as follows: it identifies the path one vertex at a time, starting from the root, and proceeding to the leaf, in $d$ rounds. In each round, given the most recently identified vertex $v_{i}$ in level $\ell$, it attempts to identify the child that vertex $v_{i}$ is labelled with. It queries every user with the same local randomizer, which asks them to use randomized response to identify the labelled child of $v_{i}$ if their data corresponds to level $\ell$, and to respond with a uniformly random child otherwise (recall that the level that a user’s data corresponds to is itself private, and hence is not known to the protocol). Since there are roughly $\tilde{\Theta}(\sqrt{n}/\varepsilon^{2})$ users with relevant data, out of $n$ users total, it is possible to identitify the child in question subject to local differential privacy. Although every user applies an $\varepsilon$-local randomizer $d$ times in sequence, because each user’s data corresponds to only a single level in the tree, the protocol is still $(\varepsilon,0)$-locally private. Note that this privacy analysis mirrors the “histogram” structure of the non-compositional protocol in Example 2.2.

Informally, the reason that any sequentially interactive protocol must have sample complexity that is larger by a factor of $d$, is that even to identify the child of a single vertex in the local model, $\Omega(d^{2}/\varepsilon^{2})$ datapoints are required (this is exactly what our randomized response protocol achieves). But a sequentially interactive protocol cannot re-use these datapoints across levels of the tree, and so must expend $\Omega(d^{2}/\varepsilon^{2})$ samples for each of the $d$ levels of the tree. This intuition is formalized in a delicate and technical induction on the depth of the tree, using information theoretic tools to bound the success probability of any protocol as a function of its sample complexity. The precise definition of $\mathcal{MPJ}(d)$ is somewhat more complicated, in which half of the weight on the underlying distribution is assigned to “level 0” dummy agents whose purpose is to break correlations between levels of the tree in the argument.

4.1 The Multi-Party Pointer Jumping Problem

We now formally define the Multi-party Pointer Jumping ($\mathcal{MPJ}$) problem.

Definition 4.1**.**

Given integer parameter $d>1$, an instance of Multi-party Pointer Jumping $\mathcal{MPJ}(d)$ is defined by a vector $Z=Z_{1}\circ\cdots\circ Z_{d}$, a concatenation of $d$ vectors of increasing length. Letting $s=d^{4}$, for each $i\in[d]$ $Z_{i}$ is a vector of $s^{i-1}$ integers in $\{0,1,\ldots,s-1\}$. For each $Z_{i}$, $Z_{i,j}$ is its $j^{th}$ coordinate.

Viewed as a tree, $Z$ is a complete $s$-ary tree of depth $d$ where each $Z_{i,j}$ marks a child of the $j$-th vertex at depth $i$. $P=P(Z)$ then denotes the vector of $d$ integers representing the unique root to leaf path down this tree through the children marked by $Z$. Formally, $P$ is defined in a recursive way: $P_{1}=Z_{1,1}$, …,$P_{i}=Z_{i,P_{1}\cdot s^{i-1}+P_{2}\cdot s^{i-2}+\cdots+P_{i-1}+1}$,…,$P_{d}=Z_{d,P_{1}\cdot s^{d-1}+P_{2}\cdot s^{d-2}+\cdots+P_{d-1}+1}$.

Finally, an instance $\mathcal{MPJ}(d)$ defines a data distribution $\mathcal{D}$. For each $x\sim\mathcal{D}$, with probability $1/2$, $x=(0,\emptyset)$ is a “dummy datapoint”, and with the remaining probability $x=(\ell,Z_{\ell})$ where $\ell$ is a level drawn uniformly at random from $[d]$. A protocol solves $\mathcal{MPJ}(d)$ if it recovers $P$ using samples from $\mathcal{D}$.

A graphical representation of $\mathcal{MPJ}(d)$ where $s=2$ appears in Figure 1 (we set $s=2$ in this figure for easier graphical representation).

4.2 An Upper Bound for Fully Interactive Mechanisms

Theorem 4.1**.**

There exists a fully interactive $(\varepsilon,0)$-locally private protocol (Algorithm 5) with sample complexity $n=O(d^{2}\log^{2}(d)(e^{\varepsilon}+1)^{2}/(e^{\varepsilon}-1)^{2})$ that, on any instance $Z$ of $\mathcal{MPJ}(d)$, correctly identifies $P(Z)$ with probability at least $1-1/d$.

Proof.

First, it is easy to check that the total sample complexity of Algorithm 5 is $n=u\cdot m=O\left(d^{2}\log^{2}(d)\cdot\frac{(e^{\varepsilon}+1)^{2}}{(e^{\varepsilon}-1)^{2}}\right).$ Privacy follows from the same line of logic used in Example 2.2: each agent sends $d$ bits in total, and at most one of these bits is not sampled uniformly at random. Therefore, the probability of an agent sending any binary string of $d$ bits is bounded between $\frac{1}{2^{d-1}}\cdot\frac{1}{1+e^{\varepsilon}}$ and $\frac{1}{2^{d-1}}\cdot\frac{e^{\varepsilon}}{1+e^{\varepsilon}}$, for any datapoint that they might hold. Algorithm 5 is therefore $(\varepsilon,0)$-locally private.

It remains to prove correctness. We first show that each group contains enough users from each level. For each group $g\in[u]$, define $X_{i,g,r}$ to be 1 if the $i$-th user in group $g$ has level $r$ and 0 otherwise. By definition, for any $r\in[d]$, $\mathbb{P}\left[X_{i,g,r}=1\right]=1/(2d)$. Therefore we have $\mathbb{E}\left[\sum_{i=1}^{m}X_{i,g,r}\right]=m/(2d)$, and by a Chernoff bound

[TABLE]

Define $W$ to be the event that for every $r\in[d]$ and $g\in[u]$, there are at least $m/(4d)$ users in group $g$ with level $r$. By a union bound, we know $\mathbb{P}\left[W\right]\geq 1-(ud)/d^{4}\geq 1-1/d^{2}$, so with high probability we have enough users in each level in each group.

We now analyze the quantities $Q_{r}$. For each $r\in[d]$, we want to show

[TABLE]

i.e. that the output $Q$ actually matches $P$. Conditioning on $Q_{1}=P_{1},\ldots,Q_{r-1}=P_{r-1}$ and $W$, $Z_{r,Q+1}=P_{r}$. Define $Y_{i,g,r}$ to be 1 if the bit sent by the $i$-th user of group $g$ is equal to the $j$-th bit of $P_{r}$ and 0 otherwise. If the $i$-th user has level $r$ then they send their bit using randomized response and $\mathbb{P}\left[Y_{i,g,r}=1\right]=\frac{e^{\varepsilon}}{e^{\varepsilon}+1}$. If the $i$-th user’s level is not $r$ then they send a uniform random bit $\mathbb{P}\left[Y_{i,g,r}=1\right]=1/2$. Since we conditioned on $W$, there are at least $m/(4d)$ users in group $g$ with level $r$. Thus

[TABLE]

Then we have

[TABLE]

Union bounding over all $u$ groups yields

[TABLE]

Putting this all together, Algorithm 5 outputs $P(Z)$ with probability at least

[TABLE]

∎

Note that Algorithm 5 is $k$-compositional only for $k\geq\Omega(d)$. The lower bound that we prove next (Theorem 4.2) shows that any sequentially interactive protocol for the same problem must have a larger sample complexity by a factor of $\tilde{\Omega}(d)=\tilde{\Omega}(k)$, showing that in general, the sample-complexity dependence that our reduction (Theorem 3.6) has on $k$ cannot be improved.

4.3 A Lower Bound for Sequentially Interactive Mechanisms

We prove our lower bound for sequentially interactive $(\varepsilon,0)$-locally private protocols. As previous work [9, 12] has established that $(\varepsilon,0)$- and $(\varepsilon,\delta)$-local privacy are approximately equivalent for reasonable parameter ranges, our lower bound also holds for sequentially interactive $(\varepsilon,\delta)$-locally private protocols. For an extended discussion of this equivalence, see Section 5.1.2.

Theorem 4.2**.**

Let $\mathcal{A}$ be a sequentially interactive $(\varepsilon,0)$-locally private protocol that, for every instance $Z$ of $\mathcal{MPJ}(d)$, correctly identifies $P(Z)$ with probability $\geq 2/3$. Then $\mathcal{A}$ must have sample complexity $n\geq d^{3}/(216(e^{\varepsilon}-1)^{2}\log(d))$.

Proof.

We will prove that any sequentially interactive $(\varepsilon,0)$-locally private protocol with $n=d^{3}/(216(e^{\varepsilon}-1)^{2}\log(d))$ samples fails to solve $\mathcal{MPJ}(d)$ correctly with probability $>1/3$ when $Z$ is sampled uniformly randomly. This is a distributional lower bound which is only stronger than the theorem statement (a worst case lower bound). For notational simplicity,we assume in this argument that all local randomizers have discrete message spaces. However, this assumption is without loss of generality and can be removed (e.g. using the rejection sampling technique from Bassily and Smith [5]).

We will prove our lower bound even for protocols to which we “reveal” some information about the hidden instance $Z$ and users’ inputs to the protocol and users. This only makes our lower bound stronger, as the mechanism can ignore this information if desired. Before the protocol starts, each user $i$ publishes a quantity $R_{i}$. $R_{i}=\ell_{i}$, user $i$’s level, if $\ell_{i}\neq 0$ (i.e., user $i$ is not a “dummy” user). Otherwise $R_{i}$ is set to be $\lfloor\frac{i-1}{n/d}\rfloor+1$. At a high level, we reveal these $\{R_{i}\}_{i=1}^{n}$ to break the dependence between $Z_{i}$’s during the execution of the protocol (see Claim 4.3 for a formalization of this intuition). Throughout the proof and its claims, we fix realizations $R_{1}=r_{1},R_{2}=r_{2},\ldots,R_{n}=r_{n}$. We will show that even given such $r_{1},...,r_{n}$, any sequentially interactive $(\varepsilon,0)$-locally private protocol with $n$ users fails with probability more than $1/3$.

For each $i\in[n]$, denote by $\Pi_{i}$ the message — i.e., portion of the transcript — sent by user $i$ via their local randomizer. Note that there is at most one such message since the protocol is sequentially interactive. We begin with a result about how conditioning on messages and revealed values affects the distribution of $Z$.

Claim 4.3**.**

Suppose $Z_{1},...,Z_{d}$ are sampled from a product distribution. Conditioned on the messages $\Pi_{1},...,\Pi_{i}$ of the first $i$ users and the revealed values $R_{1},\ldots,R_{n}$, $Z_{1},...,Z_{d}$ are still distributed according to a product distribution.

Proof.

We proceed via induction on the number of messages $i$. The base case $i=0$ is immediate from the assumption. Now suppose the statement of the claim is true for $i-1$. Use $\mathcal{D}_{1}\times\mathcal{D}_{2}\times\cdots\mathcal{D}_{d}$ to denote the product distribution of $Z_{1},...,Z_{d}$ conditioned on $\Pi_{1},...,\Pi_{i-1}$ and $R_{1},\ldots,R_{n}$ (all quantities that follow are conditioned on $R_{1},\ldots,R_{n}$, and so for notational simplicity we elide the explicit conditioning).

Since the protocol is sequentially interactive, conditioned on $\Pi_{1},...,\Pi_{i-1}$, $\Pi_{i}$ depends only on $Z_{r_{i}}$, user $i$’s internal randomness, and their level $\ell_{i}$ (recall that when $r_{i}=\lfloor\frac{i-1}{n/d}\rfloor+1$, it may be that $\ell_{i}=0$ or $\ell_{i}=r_{i}$). Therefore, conditioned on $\Pi_{1},...,\Pi_{i}$, $Z_{1},...,Z_{d}$ distribute as

[TABLE]

a product distribution. ∎

We also use induction to prove the overall theorem. It has $d$ steps. For step $\ell\in[d]$, we let $\Delta_{\ell}$ be the following set of distributions on $Z$.

Definition 4.2** ($\Delta_{\ell}$).**

For each $\ell\in[d]$, the set $\Delta_{\ell}$ is composed of distributions $\mathcal{D}$ on $Z$ such that

$\mathcal{D}$* is a product distribution on $Z_{1},...,Z_{d}$,* 2. 2.

for each $i=1,...,d-\ell$, $Z_{i}$ is deterministically fixed to be $z_{i}$, and 3. 3.

since $Z_{1},...,Z_{d-\ell}$ are fixed, by the definition of $\mathcal{MPJ}$, $P_{1},...,P_{d-\ell}$ are also fixed to some $p_{1},...,p_{d-\ell}$. The marginal distribution on $Z_{|p_{1},...,p_{d-\ell}}$ is the uniform distribution.

In the induction step, we consider locally private sequentially interactive protocols with fewer users. The idea is that for any sequentially interactive $(\varepsilon,0)$-locally private protocol of $n$ users, if we fix the messages of the first $i$ users, then what remains is a sequentially interactive $(\varepsilon,0)$-locally private protocol on $n-i$ users. Accordingly, we want to lower bound the failure probability of this remaining protocol. More concretely:

Inductive Statement

Any sequentially interactive $(\varepsilon,0)$-locally private protocol with $n\cdot\tfrac{\ell}{d}$ users (the $\left(n\cdot\tfrac{d-\ell}{d}+1\right)$-th user to the $n$-th user) fails to solve $\mathcal{MPJ}(d)$ correctly with probability at $>2/3-\ell/(3d)$ when $Z$ is sampled from a distribution in $\Delta_{\ell}$.

It will be slightly more convenient to establish the inductive case first and then establish the base case second.

Induction step ($\ell>1$):

Assume the above statement is true for $\ell-1$.

In this induction, let $\mathcal{A}$ be a sequentially interactive $(\varepsilon,0)$-locally private protocol with $n\cdot\tfrac{\ell}{d}$ users and let $\mathcal{D}$ be the distribution generating $Z$ before $\mathcal{A}$ starts. Let $\Pi$ be the messages sent by the first $n/d$ users of $\mathcal{A}$ (the $\left(n\cdot\frac{d-\ell}{d}+1\right)$-th user to the $\left(n\cdot\frac{d-\ell+1}{d}\right)$-th user) and let $\mathcal{A}_{\pi}$ be the sequentially interactive $(\varepsilon,0)$-locally private protocol with $n\cdot\frac{\ell-1}{d}$ users conditioned on $\Pi=\pi$. For notational convenience, define $n_{\ell}=n\cdot\frac{d-\ell}{d}$, $\Pi_{<i}=\Pi_{n_{\ell}+1},...,\Pi_{i-1}$ and $\Pi_{\leq i}=\Pi_{n_{\ell}+1},...,\Pi_{i}$.

For each prefix of messages, $\pi$, let $\mathcal{D}^{\prime}(\pi)$ be some mixture of distributions in $\Delta_{\ell-1}$ (to be specified later). By the induction hypothesis on $\ell-1$,

[TABLE]

Thus

[TABLE]

It therefore suffices to show that

[TABLE]

We show this via a sequence of three claims (Claims 4.4, 4.6, and 4.8), where $\mathcal{D}^{\prime}(\pi)$ is defined in Claim 4.8.

First we define some notation for the path we need to reason about. Since $\mathcal{D}\in\Delta_{\ell}$, by the definition of $\Delta_{\ell}$ we know that for $Z\sim\mathcal{D}$, the first $d-\ell$ levels of the tree $Z_{1},...,Z_{d-\ell}$ deterministically take fixed values $z_{1},...,z_{d-\ell}$. Thus, the first $d-\ell$ nodes in the path $P_{1},...,P_{d-\ell}$ are also fixed to take particular values $p_{1},...,p_{d-\ell}$. For the induction step, we write $P=P_{1},...,P_{d-\ell+1}$ to denote the first $d-\ell+1$ vertices of the path. Since $P_{d-\ell+1}$ is the only value that is not fixed, and the path is through an $s$-ary tree, $P$ can take on at most $s$ different possible values and is determined by $Z_{d-\ell+1}$.

In the first claim, we show that after observing the messages sent by $n/d$ agents, there remains substantial uncertainty about $P$.

Claim 4.4**.**

For $i\in\{n_{\ell}+1,...,n_{\ell}+n/d\}$,

[TABLE]

Proof.

Denoting by $\textbf{1}_{E}$ the indicator function for event $E$,

[TABLE]

Now consider some specific $p$. We know that

[TABLE]

For $j=n_{\ell}+1,...,i$, define random variable

[TABLE]

As we ultimately want to upper bound the quantity in Equation 2, we now focus on bounding these $X_{j}$.

Recall that $r_{j}$ is user $j$’s level if that level is non-zero (i.e. user $j$ is not a “dummy” user). Otherwise $r_{j}$ is $d-\ell+1$ for $j=n_{\ell}+1,...,n_{\ell}+n/d$. If $r_{j}\neq d-\ell+1$, by Claim 4.3, we know that conditioned on $\Pi_{<j}$, $\Pi_{j}$ is independent of $P$. Therefore when $r_{j}\neq d-\ell+1$, $X_{j}=\log(1)=0$.

If instead $r_{j}=d-\ell+1$, we know the level $\ell_{j}$ of the user $j$ is [math] with probability $d/(d+1)$ and $d-\ell+1$ with probability $1/(d+1)$. If $\ell_{j}=0$, then the user is a “dummy” and since the user has no private data about $P$, $\Pi_{j}$ is independent of $P$ conditioned on $\Pi_{<j}$. Call the input distribution of the $j$-th user $q_{j}$. Here, we recall Lemmas 3 and 4 from Duchi et al. [17] and restate a simplified version as Lemma 4.5

Lemma 4.5**.**

Let $m_{1}$ and $m_{2}$ be the output distributions of an $(\varepsilon,0)$-local randomizer in a sequentially interactive protocol given, respectively, input distributions $q_{j}\mid\Pi_{<j},P=p$ and $q_{j}\mid\Pi_{<j}$. Then

[TABLE]

We know that $\|(q_{j}|\Pi_{<j}=\pi_{<j},P=p)-(q_{j}|\Pi_{<j}=\pi_{<j})\|_{TV}\leq 1/(d+1)$, as the difference stems from the event that $\ell_{j}=d-\ell+1$. Thus, by Lemma 4.5

[TABLE]

Next, we bound the conditional expectation of $X_{j}$:

[TABLE]

Therefore $X_{n_{\ell}+1},X_{n_{\ell}+1}+X_{n_{\ell}+2},...,X_{n_{\ell}+1}+\cdots+X_{i}$ form a supermartingale. Next, we use the above bounds on these $X_{j}$ to control their sum using the Azuma-Hoeffding inequality:

[TABLE]

By the chain rule and Bayes’ rule, we know

[TABLE]

Therefore

[TABLE]

Tracing the above inequality back through Equation 2, we have

[TABLE]

∎

In Claim 4.6, we bound the information $\Pi$ contains about $Z_{|P}$ (for a primer on information theory, see Appendix B). Intuitively, by Claim 4.4 users have little information about $P$, and as a result they cannot know which potential subtree $Z_{|p}$ to focus their privacy budget on.

Claim 4.6**.**

[TABLE]

Proof.

By the inductive hypothesis, $Z$ is sampled from $\mathcal{D}\in\Delta_{\ell}$. Define $Z_{|<p}$ to be

$Z_{|p_{1},...,p_{d-\ell},0},...,Z_{|p_{1},...,p_{d-\ell},p_{d-\ell+1}-1}$. By the definition of $\Delta_{l}$, we know $Z_{|<p}$ and $Z_{|p}$ are independent given $P$, so $I(Z_{|<p};Z_{|p}|P=p)=0$. Therefore by the chain rule for mutual information, we get

[TABLE]

The main step of the proof of this claim is to compare $I(\Pi_{i};Z_{|p}|P=p,\Pi_{<i}=\pi_{<i},Z_{|<p})$ and $I(\Pi_{i};Z_{|p}|\Pi_{<i}=\pi_{<i},Z_{|<p})$. First, by Claim 4.3, conditioning on $\Pi_{<i}=\pi_{<i}$ induces a product distribution for $Z_{1},...,Z_{d}$. We also know that (as mentioned in the proof of Claim 4.3) conditioned on $\Pi_{<i}=\pi_{<i}$, $\Pi_{i}$ only depends on $Z_{r_{i}}$, the internal randomness of the user $i$, and their level $\ell_{i}$. By item 3 in the definition of $\Delta_{\ell}$, $P$ only depends on $Z_{d-\ell+1}$. We prove

[TABLE]

There are two cases depending on $r_{i}$.

• •

When $r_{i}\leq d-\ell+1$, user $i$ either has $\ell_{i}\leq d-\ell+1$ or is a “dummy” user. Therefore, whether or not we condition on $P=p$, $\Pi_{i}$ is independent of $Z_{|p},Z_{|<p}$. Thus

[TABLE]

• •

When $r_{i}>d-\ell+1$, once we’ve conditioned on $\Pi_{<i}=\pi_{<i}$, additionally conditioning on $P=p$ does not change the joint distribution of $Z_{d-\ell+2},...,Z_{d}$. This is because $P=P_{1},\ldots,P_{d-\ell+1}$ and by above conditioning on $\Pi_{<i}=\pi_{<i}$ induces a product distribution on $Z_{1},\ldots,Z_{d}$ (and in particular on $Z_{d-\ell+2},...,Z_{d}$). It follows that conditioning on $P=p$ does not change the joint distribution of $Z_{|p},Z_{|<p},\Pi_{i}$. Thus

[TABLE]

Putting things together, we have

[TABLE]

We now bound $I(\Pi_{i};Z|\Pi_{<i}=\pi_{<i})$ using Theorem 1 from Duchi et al. [17], simplified here as Lemma 4.7.

Lemma 4.7**.**

Let $\Pi$ be the distribution over randomizer outputs for an $\varepsilon$-local randomizer with inputs drawn from a distribution family parametrized by $\mathcal{V}$. Then $I(\Pi;\mathcal{V})\leq 4(e^{\varepsilon}-1)^{2}$.

In particular, the proof of Lemma 4.7 implies that $I(\Pi_{i};Z|\Pi_{<i}=\pi_{<i})\leq 4(e^{\varepsilon}-1)^{2}.$ We continue our chain of inequalities.

[TABLE]

∎

In our last claim, we convert the bound on mutual information from Claim 4.6 into a bound on the $L_{1}$ distance between distributions.

Claim 4.8**.**

There exists a distribution $\mathcal{D}^{\prime}(\pi)$ which is a mixture of distributions in $\Delta_{\ell-1}$ for each $\pi$ such that

[TABLE]

Proof.

By the definition of mutual information in terms of KL-divergence,

[TABLE]

Next, by Pinsker’s inequality,

[TABLE]

so we may upper bound

[TABLE]

On the other hand, we can also lower bound

[TABLE]

where the last equality comes from multiplying by

[TABLE]

and the inequality uses

[TABLE]

and the triangle inequality. With the preceding upper bound, the quantity in Equation 5 is $\leq 1/(3d)$.

Now, define $\mathcal{D}^{\prime}(\pi)$ to be the distribution on $Z$ such that for all $z$,

[TABLE]

Equivalently, $Z\sim\mathcal{D}^{\prime}(\pi)$ is sampled through the following procedure: (1) sample $P$ according to $P\mid\Pi=\pi$, (2) sample $Z_{|p}$ according to $Z_{\mid p}\mid P=p$, and (3) sample $Z$ according to $Z\mid\Pi=\pi,P=p,Z_{\mid p}=z_{\mid p}$.

Noting that $\mathbb{P}_{Z\sim\mathcal{D}|(\Pi=\pi)}\left[Z=z\right]=\mathbb{P}\left[Z=z|\Pi=\pi\right]$ for all $z$, since the quantity in Equation 5 is $\leq 1/(3d)$ we get

[TABLE]

It remains to show that $\mathcal{D}^{\prime}(\pi)$ is a mixture of distributions in $\Delta_{\ell-1}$; doing so will complete our proof of the original inductive step. We will show for any $z_{1},...,z_{d-\ell+1}$ such that $\mathbb{P}_{Z\sim\mathcal{D}^{\prime}(\pi)}\left[Z_{1},...,Z_{d-\ell+1}=z_{1},...,z_{d-\ell+1}\right]\neq 0$, $\mathcal{D}^{\prime}(\pi)\mid(Z_{1},...,Z_{d-\ell+1}=z_{1},...,z_{d-\ell+1})$ is a distribution in $\Delta_{\ell-1}$. Recalling that membership in $\Delta_{\ell-1}$ requires meeting three conditions, we verify these conditions below.

By Claim 4.3, we know $\mathcal{D}|(\Pi=\pi)$ is a product distribution on $Z_{1},...,Z_{d}$. It is easy to check that as $\mathcal{D}^{\prime}(\pi)$ is sampled according to $\mathcal{D}|(\Pi=\pi)$, $\mathcal{D}^{\prime}(\pi)$ is also a product distribution on $Z_{1},...,Z_{d}$, and after the conditioning, $\mathcal{D}^{\prime}(\pi)|(Z_{1},...,Z_{d-\ell+1}=z_{1},...,z_{d-\ell+1})$ remains a product distribution on $Z_{1},...,Z_{d}$. 2. 2.

Since we draw the final $Z$ conditioned on $Z_{\mid p}=z_{\mid p}$, $Z_{i}$ is deterministically fixed for $i=1,\ldots,d-\ell$. 3. 3.

First, note that the marginal distribution of $\mathcal{D}|(P=p)$ on $Z_{|p}$ is uniform since $D\mid(\Pi=\pi)$ induces a product distribution on $Z_{1},\ldots,Z_{d}$, and conditioning on $P=p$ only fixes $Z_{\leq d-\ell+1}$ and leaves $Z_{d-\ell+2}\times\cdots Z_{d}$ as a product distribution. Thus

[TABLE]

so the marginal distribution of $\mathcal{D}^{\prime}(\pi)|(Z_{1},...,Z_{d-\ell+1}=z_{1},...,z_{d-\ell+1})$ on $Z_{|p}$ is also the uniform distribution. Therefore $\mathcal{D}^{\prime}(\pi)|(Z_{1},...,Z_{d-\ell+1}=z_{1},...,z_{d-\ell+1})$ is a distribution in $\Delta_{\ell-1}$ and $\mathcal{D}^{\prime}(\pi)$ is a mixture of distributions in $\Delta_{\ell-1}$.

∎

Base case ($\ell=1$):

We finally discuss the base case of our induction. Define $\mathcal{A}$, $\Pi$ and $P$ as in the induction step. Since the output of $\mathcal{A}$ is a function of $\Pi$,

[TABLE]

Since Claim 4.4 also applies to the base case, we get

[TABLE]

∎

5 Hypothesis Testing

We now turn our attention to the role of interactivity in hypothesis testing. We first show that for the simple hypothesis testing problem, there exists a non-interactive $(\varepsilon,0)$-LDP protocol that achieves optimal sample complexity. This result extends to the compound hypothesis testing case, when we make the additional assumption that the sets of distributions are convex and compact.

5.1 Simple Hypothesis Testing

Let $P_{0}$ and $P_{1}$ be two known distributions such that $\|P_{0}-P_{1}\|_{TV}\geq\alpha$, and suppose one of $P_{0}$ and $P_{1}$ generates $n$ i.i.d. samples $x_{1},\ldots,x_{n}$. The goal in simple hypothesis testing is to determine whether the samples are generated by $P_{0}$ or $P_{1}$. The Neyman-Pearson lemma [29] establishes that the likelihood ratio test is optimal for this problem absent privacy, and recent work [10] extends this idea to give an optimal (up to constants) private simple hypothesis test in the centralized model of differential privacy. We recall a simple folklore non-interactive hypothesis test in the local model, and then prove that it is optimal even among the set of all fully interactive locally private tests.

5.1.1 (Folklore) Upper Bound

Consider the following simple variant $\mathcal{A}$ of the likelihood ratio test: each user $i$ with input $x_{i}$ outputs $\mathsf{RR}\left(\varepsilon\right){\arg\max_{j\in\{0,1\}}P_{j}(x_{i})}$. For $j\in\{0,1\}$ let $\hat{N}_{j}$ denote the resulting count of responses and let $\hat{N}_{j}^{\prime}=\tfrac{e^{\varepsilon}+1}{e^{\varepsilon}-1}\cdot\left(\hat{N}_{j}-\tfrac{n}{e^{\varepsilon}-1}\right)$ be the corresponding de-biased count. The analyst computes both quantities $\hat{N}_{j}^{\prime}$ and outputs $P_{\arg\max_{j}\hat{N}_{j}^{\prime}}$.

It is immediate that $\mathcal{A}$ is noninteractive and, since it relies on randomized response, satisfies $(\varepsilon,0)$-local differential privacy. Since we can bound its sample complexity by simple concentration arguments, we defer the proof to Appendix A.2.

Theorem 5.1**.**

With probability at least $2/3$, $\mathcal{A}$ distinguishes between $P_{0}$ and $P_{1}$ given $n=\Omega\left(\tfrac{1}{\varepsilon^{2}\alpha^{2}}\right)$ samples.

5.1.2 A Lower Bound for Arbitrarily Adaptive $(\varepsilon,\delta)$-Locally Private Tests

We now show that the folklore $\varepsilon$-private non-interactive test is optimal amongst all $(\varepsilon,\delta)$-private fully interactive tests. First, combining (slightly modified versions of) Theorem 6.1 from Bun et al. [9] and Theorem A.1 in Cheu et al. [12], we get the following result555 Bun et al. [9] and Cheu et al. [12] prove their results for noninteractive protocols. However, their constructions both rely on replacing a single $(\varepsilon,\delta)$-local randomizer call for each user with an $(O(\varepsilon),0)$-local randomizer call and proving that these randomizers induce similar output distributions. Since each user still makes a single randomizer call in sequential interactive protocols, essentially the same argument applies. For fully interactive protocols, a naive modification of the same result forces a stronger restriction on $\delta$, roughly $\delta=\tilde{o}\left(\frac{\varepsilon\beta}{\max(n,T)}\right)$.

Lemma 5.2**.**

Given $\varepsilon>0$, $\delta<\min\left(\tfrac{\varepsilon\beta}{48n\ln(2n/\beta)},\frac{\beta}{64n\ln(n/\beta)e^{7\varepsilon}}\right)$ and sequentially interactive $(\varepsilon,\delta)$-locally private protocol $\mathcal{A}$, there exists a sequentially interactive $(10\varepsilon,0)$-locally private protocol $\mathcal{A}^{\prime}$ such that for any dataset $U$, $\|\mathcal{A}(U)-\mathcal{A}^{\prime}(U)\|_{TV}\leq\beta$.

Lemma 5.2 enables us to apply existing lower bound tools for $\varepsilon$-locally private protocols to (sequentially interactive) $(\varepsilon,\delta)$-locally private protocols. At a high level, our proof relies on controlling the Hellinger distance between transcript distributions induced by an $(\varepsilon,\delta)$-locally private protocol when samples are generated by $P_{0}$ and $P_{1}$. We borrow a simulation technique used by Braverman et al. [8] for a similar (non-private) problem and find that we can control this Hellinger distance by bounding the KL divergence between a simpler, noninteractive pair of transcript distributions. We accomplish this last step using existing tools from Duchi et al. [17].

Theorem 5.3**.**

Let $\|P_{0}-P_{1}\|_{TV}=\alpha$ and let $\Pi$ be an arbitrary (possibly fully interactive) $(\varepsilon,\delta)$-locally private simple hypothesis testing protocol distinguishing between $P_{0}$ and $P_{1}$ with probability $\geq 2/3$ using $n$ samples where $\varepsilon>0$ and $\delta<\min\left(\tfrac{\varepsilon^{3}\alpha^{2}}{48n\ln(2n/\beta)},\frac{\varepsilon^{2}\alpha^{2}}{64n\ln(n/\beta)e^{7\varepsilon}}\right)$. Then $n=\Omega\left(\tfrac{1}{\varepsilon^{2}\alpha^{2}}\right).$

Proof.

Let $\Pi_{\vec{0}},\Pi_{\vec{1}}$, and $\Pi_{\vec{e}_{i}}$ respectively denote the distribution over transcripts induced by protocol $\Pi$ when samples are drawn from $P_{0}$, $P_{1}$, and $x_{i}\sim P_{1}$ but the remaining $x_{i^{\prime}}\sim P_{0}$. Let $h^{2}$ denote the square of the Hellinger distance, $h^{2}\left(f,g\right)=1-\int_{\mathcal{X}}\sqrt{f(x)g(x)}dx$. We begin with Lemma 5.4, originally proven as Lemma 2 in Braverman et al. [8].

Lemma 5.4**.**

$h^{2}\left(\Pi_{\bar{0}},\Pi_{\bar{1}}\right)=O\left(\sum_{i=1}^{n}h^{2}\left(\Pi_{\bar{0}},\Pi_{\vec{e}_{i}}\right)\right)$.

Since our goal is now to bound these squared Hellinger distances, we will use a few facts collected below.

Fact 5.5**.**

For any distributions $f$, $g$, and $h$,

$h^{2}\left(f,g\right)\leq 2(h^{2}\left(f,h\right)+h^{2}\left(h,g\right))$. 2. 2.

$h^{2}\left(f,g\right)\leq d_{TV}(f,g)\leq\sqrt{2}h\left(f,g\right)$. 3. 3.

$h^{2}\left(f,g\right)\leq\tfrac{1}{2}D_{KL}\left(f\mid\mid g\right)$.

Choose an arbitrary term $i$ of the sum in Lemma 5.4. Suppose we have user $i$ simulate $\mathcal{A}$ using draws from $P_{0}$ for the inputs of other users and their input $x_{i}$ for input $i$. Since $\Pi$ is $(\varepsilon,\delta)$-locally private, this simulation can be viewed as a single $(\varepsilon,\delta)$-local randomizer applied to $x_{i}$. We can therefore use Lemma 5.2 to get a $(10\varepsilon,0)$-local randomizer $\Pi^{\prime}$ inducing distributions $\Pi_{\vec{0}}^{\prime}$ and $\Pi_{\vec{e}_{i}}^{\prime}$ such that $\|\Pi_{\vec{0}}^{\prime}-\Pi_{\vec{0}}\|_{TV}\leq\varepsilon^{2}\alpha^{2}$ and $\|\Pi_{\vec{e}_{i}}^{\prime}-\Pi_{\vec{e}_{i}}\|_{TV}\leq\varepsilon^{2}\alpha^{2}$. Then,

[TABLE]

where the first two inequalities follow from item 1 in Fact 5.5, the third inequality follows from item 2, and the last inequality follows from our use of Lemma 5.2 above.

It remains to bound $h^{2}\left(\Pi_{\vec{0}}^{\prime},\Pi_{\vec{e}_{i}}^{\prime}\right)$. By item 3 in Fact 5.5, $4h^{2}\left(\Pi_{\vec{0}}^{\prime},\Pi_{\vec{e}_{i}}^{\prime}\right)\leq 2D_{KL}\left(\Pi_{\vec{0}}^{\prime}\mid\mid\Pi_{\vec{e}_{i}}^{\prime}\right)$. Since the transcript distributions $\Pi_{\vec{0}}^{\prime}$ and $\Pi_{\vec{e}_{i}}^{\prime}$ can be simulated by noninteractive $(10\varepsilon,0)$-local randomizers, we can apply Theorem 1 from Duchi et al. [17], restated for our setting as Lemma 5.6.

Lemma 5.6**.**

Let $Q$ be an $(\varepsilon,0)$-local randomizer and let $P_{0}$ and $P_{1}$ be distributions defined on common space $\mathcal{X}$. Let $x_{0}\sim P_{0}$ and $x_{1}\sim P_{1}$. Then

[TABLE]

Thus

[TABLE]

It follows that $h^{2}\left(\Pi_{\vec{0}}^{\prime},\Pi_{\vec{e}_{i}}^{\prime}\right)=O(\varepsilon^{2}\alpha^{2})$. Moreover, since our original choice of $i$ was arbitrary, tracing back to Lemma 5.4 yields $h^{2}\left(\Pi_{\vec{0}},\Pi_{\vec{1}}\right)=O(n\varepsilon^{2}\alpha^{2})$. By Fact 5.5, $h^{2}\left(\Pi_{\vec{0}},\Pi_{\vec{1}}\right)\geq\tfrac{1}{2}\|\Pi_{\vec{0}}-\Pi_{\vec{1}}\|_{TV}^{2}=\Omega(1)$. Thus $n=\Omega\left(\tfrac{1}{\varepsilon^{2}\alpha^{2}}\right)$. ∎

5.2 Compound Hypothesis Testing

We now extend the reasoning of Section 5 to compound hypothesis testing. Here $P_{0}$ and $P_{1}$ are replaced by (disjoint) collections of discrete hypotheses $H_{0}$ and $H_{1}$ such that

[TABLE]

The goal is to determine whether samples are generated by a distribution in $H_{0}$ or one in $H_{1}$.

Theorem 5.7**.**

Let $H_{0}$ and $H_{1}$ be convex and compact sets of distributions over ground set $X$ such that $\inf_{(P,Q)\in H_{0}\times H_{1}}\|P-Q\|_{TV}\geq\alpha$. Then there exists noninteractive $(\varepsilon,0)$-locally private protocol $\mathcal{A}$ that with probability at least $2/3$ distinguishes between $H_{0}$ and $H_{1}$ given $n=\Omega\left(\tfrac{1}{\varepsilon^{2}\alpha^{2}}\right)$ samples.

Proof.

Let $X$ be the ground set for distributions in $H_{0}$ and $H_{1}$, and consider the two-player zero-sum game

[TABLE]

Here, the sup player chooses a distribution over events, and the inf player chooses distributions in $H_{0}$ and $H_{1}$. We will use (a simplified version of) Sion’s minimax theorem [31].

Lemma 5.8** (Sion’s minimax theorem).**

For $f\colon A\times B\to\mathbb{R}$, if

for all $a\in A$ $f(a,\cdot)$ is continuous and concave on $B$, 2. 2.

for all $b\in B$ $f(\cdot,b)$ is continuous and convex on $A$, and 3. 3.

$A$* and $B$ are convex and $A$ is compact,*

then

[TABLE]

We first verify that the three conditions of Lemma 5.8 hold. Let

[TABLE]

Linearity of expectation implies that $f(\cdot,(P,Q))$ is linear in $\Delta(2^{X})$ and $f(S,\cdot)$ is linear in $H_{0}\times H_{1}$. Therefore conditions 1 and 2 hold. Moreover, since $\Delta(2^{X})$ is convex and we assumed $H_{0}$ and $H_{1}$ to be convex and compact — properties which are both closed under Cartesian product — condition 3 holds as well. As a result,

[TABLE]

and there exists fixed distribution $S$ over events such that for all $(P,Q)\in H_{0}\times H_{1}$,

[TABLE]

This leads to the following hypothesis testing protocol $\mathcal{A}$: for each $i\in[n]$, user $i$ computes $y_{i}=\mathbb{E}_{E\sim S}\left[\textbf{1}_{x_{i}\in E}\right]$ and publishes $y_{i}+\mathsf{Lap}\left(\tfrac{1}{\varepsilon}\right)$. This protocol is immediately noninteractive, and since $y_{i}\in[0,1]$, this protocol is $(\varepsilon,0)$-locally private over $\{x_{i}\}_{i=1}^{n}$. Finally, by the same analysis used to prove Theorem 5.1 (replacing concentration of randomized responses with concentration of $\mathsf{Lap}\left(1\right)$ noise [11]) it distinguishes between $H_{0}$ and $H_{1}$ with probability at least $2/3$ using $n=\Omega\left(\tfrac{1}{\varepsilon^{2}\alpha^{2}}\right)$ samples. ∎

Since Theorem 5.3 still applies, this establishes that the above non-interactive protocol is also optimal.

Appendix A Appendix

A.1 Properties of Differential Privacy

Differentially private computations enjoy two nice properties:

Theorem A.1** (Post Processing [19]).**

Let $A:\mathcal{X}^{*}\rightarrow\mathcal{O}$ be any $(\varepsilon,\delta)$-differentially private algorithm, and let $f:\mathcal{O}\rightarrow\mathcal{O^{\prime}}$ be any function. Then the algorithm $f\circ A:\mathcal{X}^{n}\rightarrow\mathcal{O}^{\prime}$ is also $(\varepsilon,\delta)$-differentially private.

Post-processing implies that, for example, every decision process based on the output of a differentially private algorithm is also differentially private.

Theorem A.2** (Basic Composition [19]).**

Let $A_{1}:\mathcal{X}^{*}\rightarrow\mathcal{O}$, $A_{2}:\mathcal{O}\times\mathcal{X}^{*}\rightarrow\mathcal{O}^{\prime}$ be such that $A_{1}$ is $(\varepsilon_{1},\delta_{1})$-differentially private, and $A_{2}(o,\cdot)$ is $(\varepsilon_{2},\delta_{2})$-differentially private for every $o\in\mathcal{O}$. Then the algorithm $A:\mathcal{X}^{*}\rightarrow\mathcal{O^{\prime}}$ defined as $A(x)=A_{2}(A_{1}(x),x)$ is $(\varepsilon_{1}+\varepsilon_{2},\delta_{1}+\delta_{2})$-differentially private.

A.2 Proof of Theorem 5.1

Proof.

For $j\in\{0,1\}$, let $N_{j}$ denote the (unknown) true count of 0 and 1 responses, i.e. $N_{j}=|\{x_{i}\mid\arg\max_{j^{\prime}\in\{0,1\}}P_{j^{\prime}}(x_{i})=j\}|$. Then for both $j$, $\mathbb{E}\left[\hat{N}_{j}\right]=\frac{N_{j}(e^{\varepsilon}-1)+n}{e^{\varepsilon}+1}$. By a Chernoff bound, with high probability $|\hat{N}_{j}-\tfrac{N_{j}(e^{\varepsilon}-1)+n}{e^{\varepsilon}+1}|=O(\sqrt{n})$. Then since $N_{j}^{\prime}=\tfrac{e^{\varepsilon}+1}{e^{\varepsilon}-1}\cdot\left(\hat{N}_{j}-\tfrac{n}{e^{\varepsilon}-1}\right)$ we get $|\hat{N}_{j}^{\prime}-N_{j}|=O\left(\frac{e^{\varepsilon+1}}{e^{\varepsilon}-1}\cdot\sqrt{n}\right)=O\left(\frac{\sqrt{n}}{\varepsilon}\right)$. It is therefore sufficient that $\tfrac{1}{\varepsilon\sqrt{n}}=O(\alpha)$ to distinguish between $P_{0}$ and $P_{1}$, which implies the claim. ∎

A.3 High Probability Sample Complexity from Theorem 3.6

We first prove a multiplicative Azuma-Hoeffding Inequality which will drive the high probability bound.

Lemma A.3** (Multiplicative Azuma-Hoeffding Inequality).**

Let $(\gamma_{t})_{t=1}^{T},\gamma_{t}\in[0,1]$ a collection of dependent random variables, and let $(\mathcal{F}_{t})_{t=1}^{T}$ a filtration such that $\sigma(\gamma_{1},\ldots\gamma_{t-1})\subset\mathcal{F}_{t-1}$. Suppose $\forall t,\mathbb{E}\left[\gamma_{t}\mid\mathcal{F}_{t-1}\right]\leq\mu_{t}$. Then if w.p. $1,\;\sum_{t}\mu_{t}\leq\mu,$ we have for any $\delta\in[e^{-3/4\mu},1]$:

[TABLE]

Proof.

By convexity of $e^{l\gamma_{t}},\gamma_{t}\in[0,1],\mathbb{E}\left[\gamma_{t}\mid\mathcal{F}_{t-1}\right]\leq\mu_{t},\;\forall t,l$:

[TABLE]

If we define $S_{j}=\sum_{t=1}^{j}\gamma_{t}$, then:

[TABLE]

Inducting on $j$, we have:

[TABLE]

For $\varepsilon>0$, taking $l=\log(1+\varepsilon),a=(1+\varepsilon)\mu$ and using Markov’s inequality:

[TABLE]

where $\phi(z)=z-(1+z)\log(1+z)$. Since $\phi(z)\leq-z^{2}/3$ for $z\in[0,3/2]$, we get:

[TABLE]

Setting $\varepsilon=\sqrt{\frac{3\log(1/\delta)}{\mu}}$ gives the desired bound. Note that the condition $\varepsilon\in[0,3/2]$ forces $\delta\geq e^{-3/4\mu}$.

∎

Proof.

There are at most $n$ users drawn in line $18$ of $\mathsf{Reduction}$, hence it suffices to bound with high probability the number of users drawn during rejection sampling steps in line $13$. For a given user $i$ drawn during a rejection sampling step, the sample complexity on rounds where $i$ is selected can be written as $\sum_{t:i_{t}=i}^{T}\gamma_{t}N_{t}$, where $\gamma_{t}\sim\mathsf{Ber}(\frac{e^{-\varepsilon_{t}}-1}{e^{-\varepsilon}-1})$, $N_{t}\stackrel{{\scriptstyle ind}}{{\sim}}\mathsf{Geom}(p_{t})$ where $p_{t}\geq\frac{e^{-2\varepsilon}}{2},\varepsilon_{t}\leq\varepsilon$ are random variables that depend on $\Pi_{<t}$. Hence the total sample complexity over the rejection sampling rounds can be written as:

[TABLE]

First consider $\sum_{t=1}^{T}\gamma_{t}$. Let $\mathcal{F}_{t}$ be the $\sigma$-algebra generated as $\mathcal{F}_{t}=\sigma(\Pi_{<t},\varepsilon_{t},(\gamma_{l})_{l=1}^{t-1})$. Then $\mathbb{E}\left[\gamma_{t}\mid\mathcal{F}_{t-1}\right]=\frac{e^{-\varepsilon_{t}}-1}{e^{-\varepsilon}-1}=\mu_{t}$. Define $\mu=\sum_{t}\mu_{t}\leq\frac{nk\varepsilon}{1-e^{-\varepsilon}}$. Hence by Lemma A.3, with probability $1-\delta/2$, for $\delta\geq 2e^{-\frac{3}{4}\mu}$:

[TABLE]

Let $E_{\gamma}$ be the above event \big{\{}\sum_{t}\gamma_{t}\leq\sqrt{3\mu\log(2/\delta)}+\mu\big{\}}. Then for any $t$,

[TABLE]

where $Z=\sum_{t=1}^{K}N_{t}^{\prime},K=\sqrt{3\mu\log(2/\delta)}+\mu$, and $N_{t}^{\prime}\stackrel{{\scriptstyle iid}}{{\sim}}\mathsf{Geom}(\frac{e^{-\varepsilon}}{2})$. Let $\mu^{\prime}=\mathbb{E}[Z]=2e^{\varepsilon}(\sqrt{3\mu\log(2/\delta)}+\mu)$. By Theorem $2.1$ in [23] for any $t\geq 1$:

[TABLE]

Setting $t=2(\frac{\log(2/\delta)2e^{\varepsilon}}{\mu^{\prime}}+1)$ gives $Z\leq 2(\log(2/\delta)2e^{\varepsilon}+\mu^{\prime})$ with probability at least $1-\delta/2$. Hence $\mathbb{P}\left[S\geq 2(\log(2/\delta)2e^{\varepsilon}+\mu^{\prime})|E_{\gamma}\right]\leq\frac{\delta}{2}$. Finally,

[TABLE]

Substituting in the expression for $\mu^{\prime}$ gives $S=O(nk+\sqrt{nk\log\frac{1}{\delta}})$ with probability $1-\delta$, as desired.

∎

Appendix B Information Theory

We briefly review some standard facts and definitions from information theory, starting with entropy. Throughout, our $\log$ is base $e$.

Definition B.1**.**

The entropy of a random variable $X$, denoted by $H(X)$, is defined as $H(X)=\sum_{x}\Pr[X=x]\log(1/\Pr[X=x])$, and the conditional entropy of random variable $X$ conditioned on random variable $Y$ is defined as $H(X|Y)=\mathbb{E}_{y}[H(X|Y=y)]$.

Next, we can use entropy to define the mutual information between two random variables.

Definition B.2**.**

The mutual information between two random variables $X$ and $Y$ is defined as $I(X;Y)=H(X)-H(X|Y)=H(Y)-H(Y|X)$, and the conditional mutual information between $X$ and $Y$ given $Z$ is defined as $I(X;Y|Z)=H(X|Z)-H(X|YZ)=H(Y|Z)-H(Y|XZ)$.

Fact B.1**.**

Let $X_{1},X_{2},Y,Z$ be random variables, we have $I(X_{1}X_{2};Y|Z)=I(X_{1};Y|Z)+I(X_{2};Y|X_{1}Z)$.

Definition B.3**.**

The Kullback-Leibler divergence between two random variables $X$ and $Y$ is defined as $D_{KL}\left(X\mid\mid Y\right)=\sum_{x}\Pr[X=x]\log(\Pr[X=x]/\Pr[Y=x])$.

Fact B.2**.**

Let $X,Y,Z$ be random variables, we have

[TABLE]

Lemma B.3** (Pinsker’s inequality).**

Let $X,Y$ be random variables,

[TABLE]

.