Optimal lower bounds for universal relation, and for samplers and finding duplicates in streams

TL;DR

This paper establishes tight lower bounds for the universal relation problem in communication complexity, leading to optimal bounds for sampling and duplicate detection in data streams, using novel proof techniques involving encoding and reductions.

Contribution

It provides the exact randomized one-way communication complexity of the universal relation problem and introduces two innovative proofs, including a new reduction from Augmented Indexing.

Findings

Lower bounds match upper bounds for the problem.

Optimal bounds for $ ext{ell}_p$-sampling in turnstile streams.

Efficient duplicate detection in streaming models.

Abstract

In the communication problem $\mathbf{UR}$ (universal relation) [KRW95], Alice and Bob respectively receive $x, y \in\{0,1\}^n$ with the promise that $x\neq y$. The last player to receive a message must output an index $i$ such that $x_i\neq y_i$. We prove that the randomized one-way communication complexity of this problem in the public coin model is exactly $\Theta(\min\{n,\log(1/\delta)\log^2(\frac n{\log(1/\delta)})\})$ for failure probability $\delta$. Our lower bound holds even if promised $\mathop{support}(y)\subset \mathop{support}(x)$. As a corollary, we obtain optimal lower bounds for $\ell_p$-sampling in strict turnstile streams for $0\le p < 2$, as well as for the problem of finding duplicates in a stream. Our lower bounds do not need to use large weights, and hold even if promised $x\in\{0,1\}^n$ at all points in the stream. We give two different proofs of our main result. The first proof demonstrates that any algorithm $\mathcal A$ solving sampling problems in turnstile streams in low memory can be used to encode subsets of $[n]$ of certain sizes into a number of bits below the information theoretic minimum. Our encoder makes adaptive queries to $\mathcal A$ throughout its execution, but done carefully so as to not violate correctness. This is accomplished by injecting random noise into the encoder's interactions with $\mathcal A$, which is loosely motivated by techniques in differential privacy. Our second proof is via a novel randomized reduction from Augmented Indexing [MNSW98] which needs to interact with $\mathcal A$ adaptively. To handle the adaptivity we identify certain likely interaction patterns and union bound over them to guarantee correct interaction on all of them. To guarantee correctness, it is important that the interaction hides some of its randomness from $\mathcal A$ in the reduction.

Equations

$$\forall x,y\in \{0,1{\}}^n,\underset{s}{P}(\mathcal{P}\text{ is correct on inputs }x,y)\ge 1-\delta ,$$

$$P(f(X,Y)=1)\le \frac{I(X;Y)+H_2(\delta )}{\log \frac{1}{\delta }},$$

$$\log \left(\genfrac{}{}{0ex}{}{n}{m}\right)-\log \left(\genfrac{}{}{0ex}{}{n}{W}\right)$$

$$=\sum _{i=1}^{m-W}\log \frac{n-W-i+1}{m-i+1}$$

$$\ge (m-W)\cdot \log \frac{n-W}{m}$$

$$\ge (m-W)\cdot \log \frac{n-m}{m}$$

$$P(f(X,Y)=1)\le \frac{I(X;Y)+H_2(\delta )}{\log \frac{1}{\delta }},$$

$$I(X;Y)\ge E(f(X,Y))\cdot \log \frac{1}{\delta }-H_2(\delta ).$$

$$H(X|Y)\le H_2(\delta )+(1-E(f(X,Y)))\cdot b+E(f(X,Y))\cdot (b-\log \frac{1}{\delta })=b+H_2(\delta )-E(f(X,Y))\cdot \log \frac{1}{\delta }.$$

$$P(S_r=T|X=x)\le \prod _{i=1}^r\frac{\left(\genfrac{}{}{0ex}{}{n_{i-1}-n_r-1}{n_{i-1}-n_i-1}\right)}{\left(\genfrac{}{}{0ex}{}{n_{i-1}-1}{n_{i-1}-n_i-1}\right)}.$$

$$\prod _{i=1}^r\frac{\left(\genfrac{}{}{0ex}{}{n_{i-1}-n_r-1}{n_{i-1}-n_i-1}\right)}{\left(\genfrac{}{}{0ex}{}{n_{i-1}-1}{n_{i-1}-n_i-1}\right)}\le \frac{2^{6K}}{\left(\genfrac{}{}{0ex}{}{m}{n_r}\right)}.$$

$$\prod _{i=1}^r\frac{\left(\genfrac{}{}{0ex}{}{n_{i-1}-n_r-1}{n_{i-1}-n_i-1}\right)}{\left(\genfrac{}{}{0ex}{}{n_{i-1}-1}{n_{i-1}-n_i-1}\right)}=\prod _{i=1}^r\frac{(n_{i-1}-n_r-1)!n_i!}{(n_{i-1}-1)!(n_i-n_r)!}=\prod _{i=1}^r\frac{n_i^{\underline{n_r}}}{(n_{i-1}-1{)}^{\underline{n_r}}}=\prod _{i=1}^r\left(\frac{n_i^{\underline{n_r}}}{n_{i-1}^{\underline{n_r}}}\cdot \frac{n_{i-1}}{n_{i-1}-n_r}\right).$$

$$\prod _{i=1}^r\frac{n_i^{\underline{n_r}}}{n_{i-1}^{\underline{n_r}}}=\frac{n_r^{\underline{n_r}}}{n_0^{\underline{n_r}}}=\frac{n_r!(n_0-n_r)!}{n_0!}=\frac{1}{\left(\genfrac{}{}{0ex}{}{n_0}{n_r}\right)}=\frac{1}{\left(\genfrac{}{}{0ex}{}{m}{n_r}\right)}.$$

$$\prod _{i=1}^r\frac{n_{i-1}}{n_{i-1}-n_r}\le \prod _{i=1}^r\frac{1}{1-\frac{m\cdot 2^{-r/K}}{m\cdot 2^{-(i-1)/K}-1}}\le \prod _{i=1}^r\frac{1}{1-\frac{m\cdot 2^{-r/K}+1}{m\cdot 2^{-(i-1)/K}}}=\prod _{j=1}^r\frac{1}{1-2^{-j/K}-\frac{2^{\frac{r-j}{K}}}{m}}.$$

$$\frac{2^{\frac{r}{K}}}{m}\le \frac{2^{\frac{R}{K}}}{m}\le \frac{1}{4K}.$$

$$\frac{1}{1-2^{-\frac{j}{K}}-\frac{2^{\frac{r-j}{K}}}{m}}\le \frac{1}{1-(1+\frac{1}{4K})2^{-\frac{j}{K}}}.$$

$$\prod _{j=1}^{2K}\frac{1}{1-2^{-j/K}}\le (8/3{)}^{2K}\cdot \frac{K^{2K}}{(2K)!}\le (8/3{)}^{2K}\cdot \frac{K^{2K}}{(2K/e{)}^{2K}}=(4e/3{)}^{2K}<2^{4K}.$$

$$\prod _{j=2K+1}^{\infty }\frac{1}{1-2^{-j/K}}\le \prod _{j=2K+1}^{\infty }\frac{1}{1-2^{-\lfloor j/K\rfloor }}\le \prod _{i=2}^{\infty }{\left(\frac{1}{1-2^{-i}}\right)}^K\le {\left(\frac{1}{1-{\sum }_{i=2}^{\infty }{2}^{-i}}\right)}^K=2^K.$$

$$A=\bigcup _{i=1}^L(\{i\}\times [u_i]\times [100^i])$$

$$S=\bigcup _{i=1}^L(\{i\}\times S_i\times [100^i]),$$

$$\begin{array}{rl}P(i=j|(R,\{\pi (j){\}}_{j\in T},\pi (S\setminus T))\text{ s.t. }\widetilde{\mathsf{Bob}}(M,1_{\pi (T)})\text{ succeeds})& \le \frac{m\cdot 100^j}{\frac{m}{2}\cdot 100^{i^{\ast }}}\\ & \le 2\cdot 100^{-(i^{\ast }-j)}\\ & \le 50^{-(i^{\ast }-j)}.\end{array}$$

$$|\mathcal{T}|$$

$$\le 2^m\cdot \prod _{i=1}^{i^{\ast }-1}\left(\genfrac{}{}{0ex}{}{m+\frac{m}{4^{i^{\ast }-i}}}{\frac{m}{2^{i^{\ast }-i}}}\right)$$

$$\le 2^m\cdot \prod _{i=1}^{i^{\ast }-1}(2e\cdot 4^{i^{\ast }-i}{)}^{\frac{m}{4^{i^{\ast }-i}}}\text{ (using }(\genfrac{}{}{0ex}{}{n}{k})\le (en/k{)}^k\text{)}$$

$$\le 2^{O(m)}\cdot 2^{m\cdot O({\sum }_{j=1}^{\infty }j{4}^{-j})}$$

$$\le 2^{O(m)}$$

$$P(\tilde{\mathcal{P}}\text{ leaves }\mathcal{T})\le \sum _{i=1}^{i^{\ast }-1}{\left(1/12\right)}^{i^{\ast }-i}<1/10.$$

$${\mathcal{E}}_{\mathcal{T}}(\pi ):={\wedge }_{T\in \mathcal{T}}{\mathcal{E}}_T(\pi )$$

$$\underset{R}{P}(\neg ({\mathcal{E}}_{\mathcal{T}}(\pi )))\le \delta \cdot |\mathcal{T}|\le 1/20$$

$$m\cdot \sum _{i=1}^{i^{\ast }-1}100^i=\frac{m}{99}\cdot (100^{i^{\ast }}-1)<\frac{m}{99}\cdot 100^{i^{\ast }}$$

$$P(\exists v,\Vert v{\Vert }_0\le 2k:{\Pi }_{k}v=0)\le \left(\genfrac{}{}{0ex}{}{n}{2k}\right)\cdot q^{2k}\cdot q^{-m}$$

$$P(\Vert a_j{\Vert }_0>16k)<{\left(\frac{e^{\frac{16k}{\mu }-1}}{(\frac{16k}{\mu }{)}^{\frac{16k}{\mu }}}\right)}^{\mu }<{\left(\frac{16k}{\mu }\right)}^{-\Omega (k)}<(e^{-Ck}{)}^{j-j^{\ast }}$$

$$\forall \delta >0,P(X>(1+\delta )EX)<{\left(\frac{e^{\delta }}{(1+\delta {)}^{1+\delta }}\right)}^{EX}.$$

$$P(\neg \mathcal{E})=P(\exists j\ge j^{\ast }:\Vert a_j{\Vert }_0>16k)<\sum _{j=j^{\ast }}^{\infty }(e^{-Ck}{)}^{j-j^{\ast }}=O(e^{-Ck}).$$

Full text

We give two different proofs of our main result. The first proof demonstrates that any algorithm $\mathcal{A}$ solving sampling problems in turnstile streams in low memory can be used to encode subsets of $[n]$ of certain sizes into a number of bits below the information theoretic minimum. Our encoder makes adaptive queries to $\mathcal{A}$ throughout its execution, but done carefully so as to not violate correctness. This is accomplished by injecting random noise into the encoder’s interactions with $\mathcal{A}$, which is loosely motivated by techniques in differential privacy. Our correctness analysis involves understanding the ability of $\mathcal{A}$ to correctly answer adaptive queries which have positive but bounded mutual information with $\mathcal{A}$’s internal randomness, and may be of independent interest in the newly emerging area of adaptive data analysis with a theoretical computer science lens. Our second proof is via a novel randomized reduction from Augmented Indexing [MNSW98] which needs to interact with $\mathcal{A}$ adaptively. To handle the adaptivity we identify certain likely interaction patterns and union bound over them to guarantee correct interaction on all of them. To guarantee correctness, it is important that the interaction hides some of its randomness from $\mathcal{A}$ in the reduction.

1 Introduction

In turnstile $\ell_{0}$-sampling, a vector $z\in\mathbb{R}^{n}$ starts as the zero vector and receives coordinate-wise updates of the form “$z_{i}\leftarrow z_{i}+\Delta$” for $\Delta\in\{-M,-M+1,\ldots,M\}$. During a query, one must return a uniformly random element from $\mathop{support}(x)=\{i:z_{i}\neq 0\}$. The problem was first defined in [FIS08], where a data structure (or “sketch”) for solving it was used to estimate the Euclidean minimum spanning tree, and to provide $\varepsilon$-approximations of a point set $P$ in a geometric space (that is, one wants to maintain a subset $S\subset P$ such that for any set $R$ in a family of bounded VC-dimension, such as the set of all axis-parallel rectangles, $||R\cap S|/|S|-|R\cap P|/|P||<\varepsilon$). Sketches for $\ell_{0}$-sampling were also used to solve various dynamic graph streaming problems in [AGM12a] and since then have been crucially used in almost all known dynamic graph streaming algorithms222The spectral sparsification algorithm of [KLM*+*14] is a notable exception., such as for: connectivity, $k$-connectivity, bipartiteness, and minimum spanning tree [AGM12a], subgraph counting, minimum cut, and cut-sparsifier and spanner computation [AGM12b], spectral sparsifiers [AGM13], maximal matching [CCHM15], maximum matching [AGM12a, BS15, Kon15, AKLY16, CCE*+*16, AKL17], vertex cover [CCHM15, CCE*+*16], hitting set, $b$-matching, disjoint paths, $k$-colorable subgraph, and several other maximum subgraph problems [CCE*+*16], densest subgraph [BHNT15, MTVV15, EHW16], vertex and hyperedge connectivity [GMT15], and graph degeneracy [FT16]. For an introduction to the power of $\ell_{0}$-sketches in designing dynamic graph stream algorithms, see the recent survey of McGregor [McG14, Section 3]. Such sketches have also been used outside streaming, such as in distributed algorithms [HPP*+*15, PRS16] and data structures for dynamic connectivity [KKM13, Wan15, GKKT15].

Given the rising importance of $\ell_{0}$-sampling in algorithm design, a clear task is to understand the exact complexity of this problem. The work [JST11] gave an $\Omega(\log^{2}n)$-bit space lower bound for data structures solving even the case $M=1$ which fail with constant probability, and otherwise whose query responses are $(1/3)$-close to uniform in statistical distance. They also gave an upper bound for $M\leq{\mathrm{poly}}(n)$ with failure probability $\delta$, which in fact gave $\min\{\|z\|_{0},\Theta(\log(1/\delta))\}$ uniform samples from the support of $z$, using space $O(\log^{2}n\log(1/\delta))$ (here $\|z\|_{0}$ denotes $|\mathop{support}(z)|$). Thus we say their data structure actually solves the harder problem of $\ell_{0}$-samplingk for $k=\Theta(\log(1/\delta))$ with failure probability $\delta$, where in $\ell_{0}$-samplingk the goal is to recover $\min\{\|z\|_{0},k\}$ uniformly random elements, without replacement, from $\mathop{support}(z)$. The upper and lower bounds in [JST11] thus match up to a constant factor for $k=1$ and $\delta$ a constant. We note though in many settings, even if the final application desires constant failure probability, $\ell_{0}$-samplingk with either failure probability $o(1)$ or $k>1$ (or both) is needed as a subroutine (see Figure 1).

Universal relation.

The work of [JST11] obtains its lower bound for $\ell_{0}$-sampling (and some other problems) via reductions from universal relation ($\mathbf{UR}$). The problem $\mathbf{UR}$ was first defined in [KRW95] and arose in connection with work of Karchmer and Wigderson on circuit depth lower bounds [KW90]. For $f:\{0,1\}^{n}\rightarrow\{0,1\}$, $D(f)$ is the minimum depth of a fan-in $2$ circuit over the basis $\{\neg,\vee,\wedge\}$ computing $f$. Meanwhile, the (deterministic) communication complexity $C(f)$ is defined as the minimum number of bits that need to be communicated in a correct protocol for Alice and Bob to solve the following communication problem: Alice receives $x\in f^{-1}(0)$ and Bob receives $y\in f^{-1}(1)$ (and hence in particular $x\neq y$), and they must both agree on an index $i\in[n]$ such that $x_{i}\neq y_{i}$. It is shown in [KW90] that $D(f)=C(f)$, where they then used this correspondence to show a tight $\Omega(\log^{2}n)$ depth lower bound for monotone circuits solving undirected $s$-$t$ connectivity. The work of [KRW95] then proposed a strategy to separate the complexity classes $\mathbf{NC}^{1}$ and $\mathbf{P}$: start with a function $f$ on $\log n$ bits requiring depth $\Omega(\log n)$, then “compose” it with itself $k=\log n/\log\log n$ times (see [KW90] for a precise definition of composition). If one could prove a strong enough direct sum theorem for communication complexity after composition, even for a random $f$, such a $k$-fold composition would yield a function that is provably in $\mathbf{P}$ (and in fact, even in $\mathbf{NC}^{2}$), but not in $\mathbf{NC}^{1}$. Proving such a direct sum theorem is still wide open, and the statement that it is true is known as the “KRW conjecture”; see for example the recent works [GMWW14, DM16] toward resolving this conjecture. As a toy problem en route to resolving it, [KRW95] suggested proving a direct sum theorem for $k$-fold composition of a particular function $\mathbf{UR}$ that they defined. That task was positively resolved in [EIRS91] (see also [HW90]).

The problem $\mathbf{UR}$ abstracts away the function $f$, and Alice and Bob are simply given $x,y\in\{0,1\}^{n}$ with the promise that $x\neq y$. The players must then agree on any index $i$ with $x_{i}\neq y_{i}$. The deterministic communication complexity of $\mathbf{UR}$ is nearly completely understood, with upper and lower bounds that match up to an additive $3$ bits, even if one imposes an upper bound on the number of rounds of communication [TZ97]. Henceforth we also consider a generalized problem $\mathbf{UR}_{k}$, where the output must be $\min\{k,\|x-y\|_{0}\}$ distinct indices on which $x,y$ differ. We also use $\mathbf{UR}^{\subset},\mathbf{UR}_{k}^{\subset}$ to denote the variants when promised $\mathop{support}(y)\subset\mathop{support}(x)$, and also Bob knows $\|x\|_{0}$. Clearly $\mathbf{UR},\mathbf{UR}_{k}$ can only be harder than $\mathbf{UR}^{\subset},\mathbf{UR}_{k}^{\subset}$, respectively.

More than twenty years after its initial introduction in connection with circuit depth lower bounds, Jowhari et al. in [JST11] demonstrated the relevance of $\mathbf{UR}$ in the randomized one-way communication model for obtaining space lower bounds for certain streaming problems, such as various sampling problems and finding duplicates in streams. In the one-way version, Bob simply needs to find such an index $i$ after a single message from Alice, and we only charge Alice’s single message’s length as the communication cost. If $\mathbf{R}^{\rightarrow,pub}_{\delta}(f)$ denotes the randomized one-way communication complexity of $f$ in the public coin model with failure probability $\delta$, [JST11] showed that the space complexity of FindDuplicate$({n})$ with failure probability $\delta$ is at least $\mathbf{R}^{\rightarrow,pub}_{\frac{7}{8}+\frac{\delta}{8}}(\mathbf{UR})$. In FindDuplicate$({n})$, one is given a length-$(n+1)$ stream of integers in $[n]$, and the algorithm must output some element $i\in[n]$ which appeared at least twice in the stream (note that at least one such element must exist, by the pigeonhole principle). The work [JST11] then showed a reduction demonstrating that any solution to $\ell_{0}$-sampling with failure probability $\delta$ in turnstile streams immediately implies a solution to FindDuplicate$({n})$ with failure probability at most $(1+\delta)/2$ in the same space (and thus the space must be at least $\mathbf{R}^{\rightarrow,pub}_{\frac{15}{16}+\frac{\delta}{16}}(\mathbf{UR})$). The same result is shown for $\ell_{p}$-sampling for any $p>0$, in which the output index should equal $i$ with probability $|x_{i}|^{p}/(\sum_{j}|x_{j}|^{p})$, and a similar result is shown even if the distribution on $i$ only has to be close to this $\ell_{p}$-distribution in variational distance (namely, the distance should be bounded away from $1$). It is then shown in [JST11] that $\mathbf{R}^{\rightarrow,pub}_{\delta}(\mathbf{UR})=\Omega(\log^{2}n)$ for any $\delta$ bounded away from $1$. The approach used though unfortunately does not provide an improved lower bound for $\delta\downarrow 0$.

Seemingly unnoticed in [JST11], we first point out here that the lower bound proof for $\mathbf{UR}$ in that work actually proves the same lower bound for the promise problem $\mathbf{UR}^{\subset}$. This observation has several advantages. First, it makes the reductions to the streaming problems trivial (they were already quite simple when reducing from $\mathbf{UR}$, but now they are even simpler). Second, a simple reduction from $\mathbf{UR}^{\subset}$ to sampling problems provides space lower bounds even in the strict turnstile model, and even for the simpler support-finding streaming problem for which when queried is allowed to return any element of $\mathop{support}(z)$, without any requirement on the distribution of the index output. Both of these differences are important for the meaningfulness of the lower bound. This is because in dynamic graph streaming applications, typically $z$ is indexed by $\binom{n}{2}$ for some graph on $n$ vertices, and $z_{e}$ is the number of copies of edge $e$ in some underlying multigraph. Edges then are not deleted unless they had previously been inserted, thus only requiring correctness for strict turnstile streams. Also, for every single application mentioned in the first paragraph of Section 1 (except for the two applications in [FIS08]), the known algorithmic solutions which we cited as using $\ell_{0}$-sampling as a subroutine actually only need a subroutine for the easier support-finding problem. Finally, third and most relevant to our current work’s main focus, the straightforward reductions from $\mathbf{UR}^{\subset}$ to the streaming problems we are considering here do not suffer any increase in failure probability, allowing us to transfer lower bounds on $\mathbf{R}^{\rightarrow,pub}_{\delta}(\mathbf{UR}^{\subset})$ for small $\delta$ to lower bounds on various streaming problems for small $\delta$. The work [JST11] could not provide lower bounds for the streaming problems considered there in terms of $\delta$ for small $\delta$.

We now show simple reductions from $\mathbf{UR}^{\subset}$ to FindDuplicate$({n})$ and from $\mathbf{UR}_{k}^{\subset}$ to support-findingk. In support-findingk we must report $\min\{k,\|z\|_{0}\}$ elements in $\mathop{support}(z)$. In the claims below, $\delta$ is the failure probability for the considered streaming problem.

Claim 1**.**

Any one-pass streaming algorithm for FindDuplicate$({n})$ must use $\mathbf{R}^{\rightarrow,pub}_{\delta}(\mathbf{UR}^{\subset})$ space.

Proof.

We reduce from $\mathbf{UR}^{\subset}$. Suppose there were a space-$S$ algorithm $\mathcal{A}$ for FindDuplicate$({n})$. Alice creates a stream consisting of all elements of $\mathop{support}(x)$ and runs $\mathcal{A}$ on those elements, then sends the memory contents of $\mathcal{A}$ to Bob. Bob then continues running $\mathcal{A}$ on $n+1-\|x\|_{0}$ arbitrarily chosen elements of $[n]\backslash\mathop{support}(y)$. Then there must be a duplicate in the resulting concatenated stream, $i$ satisfies $x_{i}\neq y_{i}$ iff $i$ is a duplicate. ∎

Claim 2**.**

Any one-pass streaming algorithm for support-findingk in the strict turnstile model must use $\mathbf{R}^{\rightarrow,pub}_{\delta}(\mathbf{UR}_{k}^{\subset})$ bits of space, even if promised that $z\in\{0,1\}^{n}$ at all points in the stream.

Proof.

This is again via reduction from $\mathbf{UR}_{k}^{\subset}$. Let $\mathcal{A}$ be a space-$S$ algorithm for support-findingk in the strict turnstile model. For each $i\in\mathop{support}(x)$, Alice sends the update $z_{i}\leftarrow z_{i}+1$ to $\mathcal{A}$. Alice then sends the memory contents of $\mathcal{A}$ to Bob. Bob then for each $i\in\mathop{support}(y)$ sends the update $z_{i}\leftarrow z_{i}-1$ to $\mathcal{A}$. Now note that $z$ is exactly the indicator vector of the set $\{i:x_{i}\neq y_{i}\}$. ∎

Claim 3**.**

Any one-pass streaming algorithm for $\ell_{p}$-sampling for any $p\geq 0$ in the strict turnstile model must use $\mathbf{R}^{\rightarrow,pub}_{\delta}(\mathbf{UR}_{k}^{\subset})$ bits of space, even if promised $z\in\{0,1\}^{n}$ at all points in the stream.

Proof.

This is via straightforward reduction from support-findingk, since reporting $\min\{k,\|z\|_{0}\}$ elements of $\mathop{support}(z)$ satisfying some distributional requirements is only a harder problem than finding any $\min\{k,\|z\|_{0}\}$ elements of $\mathop{support}(z)$. ∎

The reductions above thus raise the question: what is the asymptotic behavior of $\mathbf{R}^{\rightarrow,pub}_{\delta}(\mathbf{UR}_{k}^{\subset})$?

Our main contribution:

We prove for any $\delta$ bounded away from $1$ and $k\in[n]$, $\mathbf{R}^{\rightarrow,pub}_{\delta}(\mathbf{UR}_{k}^{\subset})=\Theta(\min\{n,t\log^{2}(n/t)\})$ where $t=\max\{k,\log(1/\delta)\}$. Given known upper bounds in [JST11], our lower bounds are optimal for FindDuplicate$({n})$, support-finding, and $\ell_{p}$-sampling for any $0\leq p<2$ for nearly the full range of $n,\delta$ (namely, for $\delta>2^{-n^{.99}}$). Also given an upper bound of [JST11], our lower bound is optimal for $\ell_{0}$-samplingk for nearly the full range of parameters $n,k,\delta$ (namely, for $t<n^{.99}$). Previously no lower bounds were known in terms of $\delta$ (or $k$). Our main theorem:

Theorem 1**.**

For any $\delta$ bounded away from $1$ and $1\leq k\leq n$, $\mathbf{R}^{\rightarrow,pub}_{\delta}(\mathbf{UR}_{k}^{\subset})=\Theta(\min\{n,t\log^{2}(n/t)\})$.

We give two different proofs of Theorem 1 (in Sections 3 and 4). Our upper bound is also new, though follows by minor modifications of the upper bound in [JST11] and thus we describe it in the appendix. The previous upper bound was $O(\min\{n,t\log^{2}n\})$. We also mention here that it is known that the upper bound for both $\mathbf{UR}_{k}$ and $\ell_{0}$-samplingk in two rounds (respectively, two passes) is only $O(t\log n)$ [JST11]. Thus, one cannot hope to extend our new lower bound to two or more passes, since it simply is not true.

1.1 Related work

The question of whether $\ell_{0}$-sampling is possible in low memory in turnstile streams was first asked in [CMR05, FIS08]. The work [FIS08] applied $\ell_{0}$-sampling as a subroutine in approximating the cost of the Euclidean minimum spanning tree of a subset $S$ of a discrete geometric space subject to insertions and deletions. The algorithm given there used space $O(\log^{3}n)$ bits to achieve failure probability $1/{\mathrm{poly}}(n)$ (though it is likely that the space could be improved to $O(\log^{2}n\log\log n)$ with a worse failure probability, by replacing a subroutine used there with a more recent $\ell_{0}$-estimation algorithm of [KNW10]). As mentioned, the currently best known upper bound solves $\ell_{0}$-samplingk using $O(t\log^{2}n)$ bits [JST11], which Theorem 1 shows is tight.

For $\ell_{p}$-sampling, conditioned on not failing, the data structure should output $i$ with probability $(1\pm\varepsilon)|x_{i}|^{p}/\|x\|_{p}^{p}$. The first work to realize its importance came even earlier than for $\ell_{0}$-sampling: [CK04] showed that an $\ell_{2}$-sampler using small memory would lead to a nearly space-optimal streaming algorithm for multiplicatively estimating $\|x\|_{3}$ in the turnstile model, but did not know how to implement such a data structure. The first implementation was given in [MW10], achieving space ${\mathrm{poly}}(\varepsilon^{-1}\log n)$ with $\delta=1/{\mathrm{poly}}(n)$. . For $1\leq p\leq 2$ the space was improved to $O(\varepsilon^{-p}\log^{3}n)$ bits for constant $\delta$ [AKO11]. In [JST11] this bound was improved to $O(\varepsilon^{-\max\{1,p\}}\log(1/\delta)\log^{2}n)$ bits for failure probability $\delta$ when $0<p<2$ and $p\neq 1$. For $p=1$ the space bound achieved by [JST11] was a $\log(1/\varepsilon)$ factor worse: $O(\varepsilon^{-1}\log(1/\varepsilon)\log(1/\delta)\log^{2}n)$ bits.

For finding a duplicate item in a stream, the question of whether a space-efficient randomized algorithm exists was asked in [Mut05, Tar07]. The question was positively resolved in [GR09], which gave an $O(\log^{3}n)$-space algorithm with constant failure probability. An improved algorithm was given in [JST11], using $O(\log(1/\delta)\log^{2}n)$ bits of space for failure probability $\delta$.

2 Overview of techniques

We now describe our two proofs of Theorem 1. For the upper bound, [JST11] achieved $O(t\log^{2}n)$, but in the appendix we show that slight modifications to their approach yield $O(\min\{n,t\log^{2}(n/t)\})$. Our main contribution is in proving an improved lower bound. Assume $t<cn$ for some sufficiently small constant $c$ (since otherwise we already obtain an $\Omega(n)$ lower bound). In both our lower bound proofs in this regime, the proof is split into two parts: we show $\mathbf{R}^{\rightarrow,pub}_{\delta}(\mathbf{UR}^{\subset})=\Omega(\log\frac{1}{\delta}\log^{2}\frac{n}{\log\frac{1}{\delta}})$ and $\mathbf{R}^{\rightarrow,pub}_{.99}(\mathbf{UR}_{k}^{\subset})=\Omega(k\log^{2}\frac{n}{k})$ separately. We give an overview the former here, which is the more technically challenging half. Our two proofs of the latter are in Sections 3.2 and 4.2.

2.1 Lower bound proof via encoding subsets and an adaptivity lemma

Our first proof of the lower bound on $\mathbf{R}^{\rightarrow,pub}_{\delta}(\mathbf{UR}^{\subset})$ is via an encoding argument. Fix $m$. A randomized encoder is given a set $S\subset[n]$ with $|S|=m$ and must output an encoding $\textsf{ENC}(S)$, and a decoder sharing public randomness with the encoder must be able to recover $S$ given only $\textsf{ENC}(S)$. We consider such schemes in which the decoder must succeed with probability $1$, and the encoding length is a random variable. Any such encoding must use $\Omega(\log(^{n}_{m}))=\Omega(m\log\frac{n}{m})$ bits in expectation for some $S$.

There is a natural, but sub-optimal approach to using a public-coin one-way protocol $\mathcal{P}$ for $\mathbf{UR}^{\subset}$ to devise such an encoding/decoding scheme. The encoder pretends to be Alice with input $x$ being the indicator set of $S$, then lets $\textsf{ENC}(S)$ be the message $M$ Alice would have sent to Bob. The decoder attempts to recover $S$ by iteratively pretending to be Bob $m$ times, initially pretending to have input $y=0\in\{0,1\}^{n}$, then iteratively adding elements found in $S$ to $y$’s support. Henceforth let $\mathbf{1}_{T}\in\{0,1\}^{n}$ denote the indicator vector of a set $T\subset[n]$.

One might hope to say that if the original failure probability were $\delta<1/m$, then by a union bound, with constant probability every iteration succeeds in finding a new element of $S$ (or one could even first apply some error-correction to $x$ so that the decoder could recover $S$ even if only a constant fraction of iterations succeeded). The problem with such thinking though is that this decoder chooses $y$’s adaptively! To be specific, $\mathcal{P}$ being a correct protocol means

[TABLE]

where $s$ is the public random string that both Alice and Bob have access to. The issue is that even in the second iteration (when $r=2$), Bob’s “input” $\mathbf{1}_{T}$ depends on $s$, since $T$ depends on the outcome of the first iteration! Thus the guarantee of (1) does not apply.

One way around the above issue is to realize that as long as every iteration succeeds, $T$ is always a subset of $S$. Thus it suffices for the following event $\mathcal{E}$ to occur: $\forall T\subset S,\ \mathcal{P}\text{ is correct on inputs }\mathbf{1}_{S},\mathbf{1}_{T}$. Then $\operatorname*{\mathbb{P}}_{s}(\neg\mathcal{E})\leq 2^{m}\delta$ by a union bound, which is at most $1/2$ for $m=\lfloor\log_{2}(1/\delta)\rfloor-1$. We have thus just shown that $\mathbf{R}^{\rightarrow,pub}_{\delta}(\mathbf{UR}^{\subset})=\Omega(\min\{n,\log(^{n}_{m})\})=\Omega(\min\{n,\log\frac{1}{\delta}\log\frac{n}{\log(1/\delta)}\})$.

Our improvement is as follows. Our new decoder again iteratively tries to recover elements of $S$ as before. We will give up though on having $m$ iterations and hoping for all (or even most) of them to succeed. Instead, we will only have $R=\Theta(\log\frac{1}{\delta}\log\frac{n}{\log\frac{1}{\delta}})$ iterations, and our aim is for the decoder to succeed in finding a new element in $S$ for at least a constant fraction of these $R$ iterations. Simplifying things for a moment, let us pretend for now that all $R$ iterations do succeed in finding a new element. $\textsf{ENC}(S)$ will then be Alice’s message $M$, together with the set $B\subset S$ of size $m-R$ not recovered during the $R$ rounds, explicitly written using $\lceil\log{n\choose|B|}\rceil$ bits. If the decoder can then recover these $R$ remaining elements, this then implies the decoder has recovered $S$, and thus we must have $|M|=\Omega(\log{n\choose m}-\log{n\choose|B|})=\Omega(R\log\frac{n}{m})$. The decoder proceeds as follows. Just as before, initially the decoder starts with $T=\emptyset$ and lets $i$ be the output of Bob on $\mathbf{1}_{T}$ and adds it to $T$. Then in iteration $r$, before proceeding to the next iteration, the decoder randomly picks some elements from $B$ and adds them into $T$, so that the number of elements left to be uncovered is some fixed number $n_{r}$. These extra elements being added to $T$ should be viewed as “random noise” to mask information about the random string $s$ used by $\mathcal{P}$, an idea very loosely inspired by ideas in differential privacy. For intuition, as an example suppose the iteration $r=1$ succeeds in finding some $i\in S$. If the decoder were then to add $i$ to $T$, as well as $\approx m/2$ random elements from $B$ to $T$, then the resulting $T$ reveals only $\approx 1$ bit of information about $i$ (and hence about $s$). This is as opposed to the $\log m$ bits $T$ could have revealed if the masking were not performed. Thus the next query in round $r=2$, although correlated with $s$, has very weak correlation after masking and we thus might hope for it to succeed. This intuition is captured in the following lemma, which we prove in Section 3.1:

Lemma 1**.**

Consider $f$: $\{0,1\}^{b}\times\{0,1\}^{q}\rightarrow\{0,1\}$ and $X\in\{0,1\}^{b}$ uniformly random. If $\forall y\in\{0,1\}^{q},\ \operatorname*{\mathbb{P}}(f(X,y)=1)\leq\delta$ where $0<\delta<1$, then for any random variable $Y$ supported on $\{0,1\}^{q}$,

[TABLE]

where $I(X;Y)$ is the mutual information between $X$ and $Y$, and $H_{2}$ is the binary entropy function.

Fix some $x\in\{0,1\}^{n}$. One should imagine here that $f(X,y)$ is $1$ iff $\mathcal{P}$ fails when Alice has input $x$ and Bob has input $y$ in a $\mathbf{UR}^{\subset}$ instance, and the public random string is $X=s$. Then the lemma states that if $y=Y$ is not arbitrary, but rather random (and correlated with $X$), then the failure probability of the protocol is still bounded as long as the mutual information between $X$ and $Y$ is bounded. It is also not hard to see that this lemma is sharp up to small additive terms. Consider the case $x,y\in[n]$, and $f(x,y)=1$ iff $x=y$. Then if $X$ is uniform, for all $y$ we have $\operatorname*{\mathbb{P}}(f(X,y)=1)=1/n$. Now consider the case where $Y$ is random and equal to $X$ with probability $t/\log n$ and is uniform in $[n]$ with probability $1-t/\log n$. Then in expectation $Y$ reveals $t$ bits of $X$, so that $I(X;Y)=t$. It is also not hard to see that $\operatorname*{\mathbb{P}}(f(X,Y)=1)\approx t/\log n+1/n$.

In light of the strategy stated so far and Lemma 1, the path forward is clear: at each iteration $r$, we should add enough random masking elements to $T$ to keep the mutual information between $T$ and all previously added elements below, say, $\frac{1}{2}\log\frac{1}{\delta}$. Then we expect a constant fraction of iterations to succeed. The encoder knows which iterations do not succeed since it shares public randomness with the decoder (and can thus simulate it), so it can simply tell the decoder which rounds are the failed ones, then explicitly include in $M$ correct new elements of $S$ for the decoder to use in the place of Bob’s wrong output in those rounds. A calculation shows that if one adds a $(1-1/K)\approx 2^{-1/K}$ fraction of the remaining items in $S$ to $T$ after drawing one more support element from Bob, the mutual information between the next query to Bob and the randomness used by $\mathcal{P}$ will be $O(K)$ (see Lemma 5). Thus we do this for $K$ a sufficiently small constant times $\log\frac{1}{\delta}$. We will then have $n_{r}\approx(1-1/K)^{r}m$. Note that we cannot continue in this way once $n_{r}<K$ (since the number of “random noise” elements we inject should at least be one). Thus we are forced to stop after $R=\Theta(K\log(m/K))=\Theta(\log\frac{1}{\delta}\log\frac{n}{\log\frac{1}{\delta}})$ iterations. We then set $m=\sqrt{n\log(1/\delta)}$, so that $\mathbf{R}^{\rightarrow,pub}_{\delta}(\mathbf{UR}^{\subset})=\Omega(|R|\log\frac{n}{m})=\Omega(\min\{n,\log\frac{1}{\delta}\log^{2}\frac{n}{\log\frac{1}{\delta}}\})$ as desired.

The argument for lower bounding $\mathbf{R}^{\rightarrow,pub}_{\delta}(\mathbf{UR}_{k}^{\subset})$ is a bit simpler, and in particular does not need rely on Lemma 1. Both the idea and rigorous argument can be found in Section 3.2, but again the idea is to use a protocol for this problem to encode appropriately sized subsets of $[n]$.

As mentioned above, our lower bounds use protocols for $\mathbf{UR}^{\subset}$ and $\mathbf{UR}^{\subset}_{k}$ to establish protocols for encoding subsets of some fixed size $m$ of $[n]$. These encoders always consist of some message $M$ Alice would have sent in a $\mathbf{UR}^{\subset}$ or $\mathbf{UR}^{\subset}_{k}$ protocol, together with a random subset $B\subset S$ (using $\lceil\log_{2}|B|\rceil+\lceil\log{n\choose|B|}\rceil$ bits, to represent both $|B|$ and the set $B$ itself). Here $|B|$ is a random variable. These encoders are thus Las Vegas: the length of the encoding is a random variable, but the encoder/decoder always succeed in compressing and recovering the subset. The final lower bounds then come from the following simple lemma, which follows from the source coding theorem.

Lemma 2**.**

Let s denote the number of bits used by the $\mathbf{UR}^{\subset}$ or $\mathbf{UR}^{\subset}_{k}$ protocol, and let $\textsf{s}^{\prime}$ denote the expected number of bits to represent $B$. Then $(1+\textsf{s}+\textsf{s}^{\prime})\geq\log(^{n}_{m})$. In particular, $s\geq\log(^{n}_{m})-s^{\prime}-1$.

Section 3.1 provides our first proof that $\mathbf{R}^{\rightarrow,pub}_{\delta}(\mathbf{UR}^{\subset})=\Omega(\min\{n,\log^{2}(\frac{n}{\log(1/\delta)})\log\frac{1}{\delta}\})$. We extend our results in Section 3.2 to $\mathbf{UR}_{k}^{\subset}$ for $k\geq 1$, proving a lower bound of $\Omega(k\log^{2}(n/k))$ communication even for constant failure probability.

2.2 Lower bound proof via reduction from $\mathbf{AugIndex}_{N}$

Our second lower bound proof for $\mathbf{UR}^{\subset}$ is via a randomized reduction from $\mathbf{AugIndex}_{N}$ [MNSW98]. In this problem, Charlie receives $z\in\{0,1\}^{N}$ and Diane receives $j^{*}\in[N]$ together with $z_{j}$ for $j=j^{*}+1,\ldots,N$, and Diane must output $z_{j^{*}}$. It is shown in [MNSW98] that $\mathbf{R}^{\rightarrow,pub}_{\delta}(\mathbf{AugIndex}_{N})=\Omega(N)$ for any $\delta$ bounded away from $1/2$. In our reduction, $N=\Theta(\log(1/\delta)\log^{2}\frac{n}{\log(1/\delta)})$.

For $\mathbf{UR}^{\subset}$, we can also think of the problem as Alice being given $S\subseteq[n]$ and Bob being given $T\subsetneq S$, and Bob must output some element of $S\backslash T$. In $\mathbf{AugIndex}_{N}$, Charlie views his input as $L=\Theta(\log\frac{n}{\log(1/\delta)})$ blocks of bits of nearly equal size, where the $i$th block represents a subset $S_{i}$ of $[u_{i}]$ in some collection $\mathcal{S}_{u_{i},m}$ of sets, for some carefully chosen universe sizes $u_{i}$ per block. Here $\mathcal{S}_{u_{i},m}$ is a collection of subsets of $[u_{i}]$ of size $m$ of maximal size such any two sets in the collection have intersection size strictly less than $m/2$. Furthermore, Diane’s index $j^{*}$ is in some particular block of bits corresponding to some set $S_{i^{*}}$, and Diane also knows $S_{i}$ for $i>j$.

Now we explain the reduction. We assume some protocol $\mathcal{P}$ for $\mathbf{UR}^{\subset}$, and we give a protocol $\mathcal{P}^{\prime}$ for $\mathbf{AugIndex}_{N}$. First, we define the universe $A=\bigcup_{i=1}^{L}(\{i\}\times[u_{i}]\times[100^{i}])$, which has size $n$. Charlie then defines $S=\bigcup_{i=1}^{L}(\{i\}\times S_{i}\times[100^{i}])$. Charlie and Diane use public randomness to define a uniformly random permutation $\pi$ on $[n]$. Charlie can compute $\pi(S)$. Also, since Diane knows $S_{i}$ for $i>i^{*}$, she can define $T=\bigcup_{i=i^{*}+1}^{L}(\{i\}\times S_{i}\times[100^{i}])$ and compute $\pi(T)$. Then $\pi(S)$ and $\pi(T)$ are the inputs to Alice and Bob in the protocol $\mathcal{P}$ for $\mathbf{UR}^{\subset}$. Charlie sends Diane the message Alice would have sent Bob in $\mathcal{P}$ if her input had been $\pi(S)$, and Diane simulates Bob to recover an element in $\pi(S)\backslash\pi(T)$. Importantly, Alice and Bob do not know anything about $\pi$ at this point other than that $\pi(S)=S$ and $\pi(T)=T$. Thus, the protocol $\mathcal{P}$ for $\mathbf{UR}^{\subset}$, if it succeeds, outputs an arbitrary element $j\in\pi(S)\backslash\pi(T)$, which is a deterministic function of the labels of elements in $\pi(S)$ and $\pi(T)$ and the randomness $R$ that Alice and Bob share, which is independent from the randomness in $\pi$. Since $\pi$ is still a uniformly random map conditioned on $\pi(S)=S$ and $\pi(t)=t$ for each $t\in T$, and $j\in\pi(S)\backslash\pi(T)$, it follows that $\pi^{-1}(t)$ is a uniformly random element of $S\setminus T$. After receiving $\pi^{-1}(j)$, if $(i,a,r)=\pi^{-1}(j)$, then Charlie and Diane reveal the pairs $((i,a,z),\pi((i,a,z)))$ for each $z\in[100^{i}]$ to Alice and Bob and Bob updates his set $\pi(T)$ to include $\pi(i,a,z)$ for each $z\in[100^{i}]$. One can show that at each step in this process, if Alice and Bob succeed in outputting an arbitrary item $j$ from $\pi(S)\setminus\pi(T)$, then this is a uniformly random item from $\pi(S)\setminus\pi(T)$. The fact that this item is uniformly random is crucial for arguing the number of computation paths of the protocol of Alice and Bob is $o(1/\delta)$ with good probability, over $\pi$, so that one can argue (see below) that with good probability on every such computation path Alice and Bob succeed on that path, over their randomness $R$. Although the idea of using a random permutation appeared in [JST11] to show that any public coin $\mathbf{UR}$ protocol can be made into one in which a uniformly random element of $S\backslash T$ is output, here we must use this idea adaptively, slowly revealing information about $\pi$ and arguing that this property is maintained for each of Bob’s successive queries.

Due to geometrically increasing repetitions of items for increasing $i$, a uniformly random element in $S\backslash T$ is roughly $100$ times more likely to correspond to an item in $S_{i^{*}}$ than in $S_{i}$ for $i<i^{*}$. Thus if Diane simulates Bob to recover a random element in $S\backslash T$, it is most likely to recover an element $j$ of $S_{i^{*}}$. She can then tell Bob to include $\pi(j)$ and its $100^{i^{*}}$ redundant copies to $\pi(T)$ and iterate.

There are several obstacles to overcome to make this work. First, iterating means using $\mathcal{P}$ adaptively, which was the same issue that arose in Section 2.1. Second, a constant fraction of the time ($1/100$), we expect to obtain an element not in $S_{i^{*}}$, but rather from some $S_{i}$ for $i<i^{*}$. If this happened too often, then Diane would need to execute many queries to recover a sufficiently large number of elements from $S_{i^{*}}$ in order to solve $\mathbf{AugIndex}_{N}$. This would then require a union bound over too many possible computation paths, which would not be possible as Alice likely would fail on one of them (over the choice of $R$). However, since the random permutation argument above ensures that at each step we receive a uniformly random item from the current set $S\setminus T$, if we continue for $m$ iterations, we can argue that with large probability, our sequence of inputs $T$ over the iterations with which Diane invokes Bob’s output are all likely to come from a family $\mathcal{T}$ of size at most $2^{O(m)}$. Here we need to carefully construct this family to contain a smaller number of sets from levels $i$ for which $i^{*}-i$ is larger so that the overall number of sets is small. Given this, we can union bound over all such $T$, for total failure probability $\delta|\mathcal{T}|\ll 1$. Furthermore, we can also argue that after $m$ iterations, it is likely that we have recovered at least $m/2$ of the elements from $S_{i^{*}}$, which is enough to uniquely identify $S_{i^{*}}\in\mathcal{S}_{u_{i},m}$ by the limited intersection property of $\mathcal{S}_{u_{i},m}$.

3 Lower bounds via the adaptivity lemma

3.1 Communication Lower Bound for $\mathbf{UR}^{\subset}$

Consider a protocol $\mathcal{P}$ for $\mathbf{UR}^{\subset}$ with failure probability $\delta$, operating in the one-way public coin model. When Alice’s input is $x$ and Bob’s is $y$, Alice sends $\mathsf{Alice}(x)$ to Bob, and Bob outputs $\mathsf{Bob}(\mathsf{Alice}(x),y)$, which with probability at least $1-\delta$ is in $\mathop{support}(x-y)$. As mentioned in Section 2, we use $\mathcal{P}$ as a subroutine in a scheme for encoding/decoding elements of $\binom{[n]}{m}$ for $m=\lfloor\sqrt{n\log(1/\delta)}\rfloor$. We assume $\log\frac{1}{\delta}\leq n/64$, since for larger $n$ we have an $\Omega(n)$ lower bound.

3.1.1 Encoding/decoding scheme

We now describe our encoding/decoding scheme $(\textsf{ENC},\textsf{DEC})$ for elements in ${[n]\choose m}$, which uses $\mathcal{P}$ in a black-box way. The parameters shared by ENC and DEC are given in Algorithm 2.

As discussed in Section 2, on input $S\in{[n]\choose m}$, ENC computes $M\leftarrow\mathsf{Alice}(\mathbf{1}_{S})$ as part of its output. Moreover, ENC also outputs a subset $B\subseteq S$ computed as follows. Initially $B=S$ and $S_{0}=S$. ENC proceeds in $R$ rounds. In round $r\in[R]$, ENC computes $s_{r}\leftarrow\mathsf{Bob}(M,\mathbf{1}_{S\backslash S_{r-1}})$. Let $b$ denote a binary string of length $R$, where $b_{r}$ records whether $\mathsf{Bob}$ succeeds in round $r$. ENC also outputs $b$. If $s_{r}\in S_{r-1}$, i.e. $\mathsf{Bob}(M,\mathbf{1}_{S\backslash S_{r-1}})$ succeeds, ENC sets $b_{r}=1$ and removes $s_{r}$ from $B$ (since the decoder can recover $s_{r}$ from the $\mathbf{UR}^{\subset}$-protocol, ENC does not need to include it in $B$); otherwise ENC sets $b_{r}=0$. At the end of round $r$, ENC picks a uniformly random set $S_{r}$ in $\binom{S_{r-1}\backslash\{s_{r}\}}{n_{r}}$. In particular, ENC uses its shared randomness with DEC to generate $S_{r}$ in such a way that $\textsf{ENC},\textsf{DEC}$ agree on the sets $S_{r}$ (DEC will actually iteratively construct $C_{r}=S\backslash S_{r}$). We present ENC in Algorithm 3.

The decoding process is symmetric. Let $C_{0}=\emptyset$ and $A=\emptyset$. DEC proceeds in $R$ rounds. On round $r\in[R]$, DEC obtains $s_{r}\in S\backslash C_{r-1}$ by invoking $\mathsf{Bob}(M,\mathbf{1}_{C_{r-1}})$. By construction of $C_{r-1}$ (to be described later), it is guaranteed that $S_{r-1}=S\backslash C_{r-1}$. Therefore, DEC recovers exactly the same $s_{r}$ as ENC. DEC initially assigns $C_{r}\leftarrow C_{r-1}$. If $b_{r}=1$, DEC adds $s_{r}$ to both $A$ and $C_{r}$. At the end of round $r$, DEC inserts many random items from $B$ into $C_{r}$ so that $C_{r}=S\backslash S_{r}$. DEC can achieve this because of the shared random permutation $\pi$ when constructing $S_{r}$. In the end, DEC outputs $B\cup A$. We present DEC in Algorithm 4.

3.1.2 Analysis

We have two random objects in our encoding/decoding scheme: (1) the random source used by $\mathcal{P}$, denoted by $X$, and (2) the random permutation $\pi$. These are independent.

First, we can prove that $\textsf{DEC}(\textsf{ENC}(S))=S$. That is, for any fixing of the randomness in $X$ and $\pi$, DEC will always decode $S$ successfully. It is because ENC and DEC share $X$ and $\pi$, so that DEC essentially simulates ENC. We formally prove this by induction in Lemma 3.

Now our goal is to prove that by using the $\mathbf{UR}^{\subset}$-protocol, the number of bits that ENC saves in expectation over the naive $\lceil\log(^{n}_{m})\rceil$-bit encoding is $\Omega(\log\frac{1}{\delta}\log^{2}\frac{n}{\log(1/\delta)})$ bits. Intuitively, it is equivalent to prove the number of elements that ENC saves is $\Omega(\log\frac{1}{\delta}\log\frac{n}{\log(1/\delta)})$. We formalize this in Lemma 4. Note that ENC also needs to output $b$ (i.e., whether the $\mathsf{Bob}$ succeeds on $R$ rounds), which takes $R$ bits. By our setting of parameters, we can afford the loss of $R$ bits. Thus it is sufficient to prove $\operatorname*{\mathbb{E}}|B|=|S|-\Omega(\log\frac{1}{\delta}\log\frac{n}{\log(1/\delta)})$.

We have $|S|-|B|=\sum_{r=1}^{R}b_{r}$. In Lemma 1, we prove the probability that $\mathsf{Bob}$ fails on round $r$ is upper bounded by $\frac{I(X;S_{r-1})+1}{\log\frac{1}{\delta}}$, where $I(X;S_{r-1})$ is the mutual information between $X$ and $S_{r-1}$. Furthermore, we will show in Lemma 5 that $I(X;S_{r-1})$ is upper bounded by $O(K)$. By our setting of parameters, we have $\operatorname*{\mathbb{E}}b_{r}=\Omega(1)$ and thus $\operatorname*{\mathbb{E}}(|S|-|B|)=\Omega(R)=\Omega(\log\frac{1}{\delta}\log\frac{n}{\log(1/\delta)})$.

Lemma 3**.**

$\textsf{DEC}(\textsf{ENC}(S))=S$.

Proof.

We claim that for $r=0,\ldots,R$, $\{S_{r},C_{r}\}$ is a partition of $S$ ($S_{r}$ is defined in Algorithm 3, and $C_{r}$ in Algorithm 4). We prove the claim by induction on $r$. Our base case is $r=0$, for which the claim holds since $S_{0}=S$, $C_{0}=\emptyset$.

Assume the claim holds for $r-1$ ($1\leq r\leq R$), and we consider round $r$. On round $r$, by induction $S\backslash S_{r-1}=C_{r-1}$, the index $s_{r}$ obtained by both ENC and DEC are the same. Initially $S_{r}=S_{r-1}$ and $C_{r}=C_{r-1}$, and so $\{S_{r},C_{r}\}$ is a partition of $S$. If $s_{r}$ is a valid sample (i.e. $s_{r}\in S_{r-1}$), then $b_{r}=1$, and ENC removes $s_{r}$ from $S_{r}$ and in the meanwhile DEC inserts $s_{r}$ into $C_{r}$, so that $\{S_{r},C_{r}\}$ remains a partition of $S$. Next, ENC repeats removing the $a$ from $S_{r}$ with the smallest $\pi_{a}$ value until $|S_{r}|=n_{r}$. Symmetrically, DEC repeats inserting the $a$ into $C_{r}$ with the smallest $\pi_{a}$ value among $a\in B\backslash C_{r}$, until $|C_{r}|=|S|-n_{r}$. In the end we have $|S_{r}|+|C_{r}|=|S|$, so ENC and DEC execute repetition the same number of times. Moreover, we can prove that during the same iteration of this repeated insertion, the element removed from $S_{r}$ is exactly the same element inserted to $C_{r}$. This is because in the beginning of a repetition $\{S_{r},C_{r}\}$ is a partition of $S$. We have $B\backslash C_{r}\subseteq S\backslash C_{r}=S_{r}$. Let $a^{*}$ denote $a\in S_{r}$ that minimizes $\pi_{a}$. Then $a^{*}\in B\backslash C_{r}\subseteq S_{r}$ (since $a^{*}$ will be removed from $S_{r}$, it has no chance to be included in $S$ in ENC, so that $B$ contains $a^{*}$), and $\pi_{a^{*}}$ is also the smallest among $\{\pi_{a}:a\in B\backslash C_{r}\}$. Thus both ENC and DEC will take $a^{*}$ (for ENC, to remove from $S_{r}$, and for DEC, to insert into $C_{r}$). Therefore, $\{S_{r},C_{r}\}$ remains a partition of $S$.

Given the fact that $\{S_{r},C_{r}\}$ is a partition of $S$, the $s_{r}$ are the same in ENC and DEC. Furthermore, $A=\{s_{r}:b_{r}=1,r=1,\ldots,R\}$ are the same in ENC and DEC. We know $A\subseteq S$. Since ENC outputs $S\backslash A$, and DEC outputs $(S\backslash A)\cup A$, we have $\textsf{DEC}(\textsf{ENC}(S))=S$. ∎

Lemma 4**.**

Let $W\in\mathbb{N}$ be a random variable with $W\leq m$ and $\operatorname*{\mathbb{E}}W\leq m-d$. Then $\operatorname*{\mathbb{E}}(\log{n\choose m}-\log{n\choose W})\geq d\log(\frac{n}{m}-1)$.

Proof.

[TABLE]

Taking expectation on both sides, we have $\operatorname*{\mathbb{E}}(\log{n\choose m}-\log{n\choose W})\geq d\log(\frac{n}{m}-1)$. ∎

Lemma 1 (restated). Consider $f$: $\{0,1\}^{b}\times\{0,1\}^{q}\rightarrow\{0,1\}$ and $X\in\{0,1\}^{b}$ uniformly random. If $\forall y\in\{0,1\}^{q},\ \operatorname*{\mathbb{P}}(f(X,y)=1)\leq\delta$ where $0<\delta<1$, then for any r.v. $Y$ supported on $\{0,1\}^{q}$,

[TABLE]

where $I(X;Y)$ is the mutual information between $X$ and $Y$, and $H_{2}$ is the binary entropy function.

Proof.

It is equivalent to prove

[TABLE]

By definition of mutual entropy $I(X;Y)=H(X)-H(X|Y)$, where $H(X)=b$ and we must show

[TABLE]

The upper bound for $H(X|Y)$ is obtained by considering the following one-way communication problem: Alice knows both $X$ and $Y$ while Bob only knows $Y$, and Alice must send a single message to Bob so that Bob can recover $X$. The expected message length in an optimal protocol is exactly $H(X|Y)$. Thus, any protocol gives an upper bound for $H(X|Y)$, and we simply take the following protocol: Alice prepends a $1$ bit to her message iff $f(X,Y)=1$ (taking $H_{2}(\delta)$ bits in expectation). Then if $f(X,Y)=0$, Alice sends $X$ directly (taking $b$ bits). Otherwise, when $f(X,Y)=1$, Alice sends the index of $X$ in $\{x|f(x,Y)=1\}$ (taking $\log(\delta 2^{b})=b-\log\frac{1}{\delta}$ bits). ∎

Corollary 1**.**

Let $X$ denote the random source used by the $\mathbf{UR}^{\subset}$-protocol with failure probability at most $\delta$. If $S$ is a fixed set and $T\subset S$, $\operatorname*{\mathbb{P}}(\mathsf{Bob}(\mathsf{Alice}(\mathbf{1}_{S}),\mathbf{1}_{T})\not\in S\backslash T)\leq\frac{I(X;T)+H_{2}(\delta)}{\log\frac{1}{\delta}}$.

Lemma 5**.**

$I(X;S_{r})\leq 6K$, for $r=1,\ldots,R$.

Proof.

Note that $I(X;S_{r})=H(S_{r})-H(S_{r}|X)$. Since $|S_{r}|=n_{r}$ and $S_{r}\subseteq S$, $H(S_{r})\leq\log{m\choose n_{r}}$. Here is the main idea to lower bound $H(S_{r}|X)$: By definition of conditional entropy, $H(S_{r}|X)=\sum_{x}{p_{x}\cdot H(S_{r}|X=x)}$. We fix an arbitrary $x$. If we can prove that for any $T\subseteq S$ where $|T|=n_{r}$, $\operatorname*{\mathbb{P}}(S_{r}=T|X=x)\leq p$, then by definition of entropy we have $H(S_{r}|X=x)\geq\log\frac{1}{p}$.

First we can prove for any fixed $T$,

[TABLE]

We have $\operatorname*{\mathbb{P}}(S_{r}=T|X=x)=\Pi_{i=1}^{r}{\operatorname*{\mathbb{P}}(T\subseteq S_{i}|T\subseteq S_{i-1})}$. On round $i$ ($1\leq i\leq r$), ENC removes $n_{i-1}-n_{i}$ elements (at least $n_{i-1}-n_{i}-1$ of which are chosen all at random) from $S_{i-1}$ to obtain $S_{i}$. Conditioned on the event that $T\subseteq S_{i-1}$, the probability that $T\subseteq S_{i}$ is at most ${{n_{i-1}-n_{r}-1\choose n_{i-1}-n_{i}-1}}/{{n_{i-1}-1\choose n_{i-1}-n_{i}-1}}$, where the equation achieves when $s_{i}\in S_{i-1}\backslash T$, and ENC takes a uniformly random subset of $S_{i-1}\backslash\{s_{i}\}$ of size $n_{i-1}-n_{i}-1$, so that the subset does not intersect with $T$.

Next we can prove

[TABLE]

For notational simplicity, let $n^{\underline{k}}$ denote $n\cdot(n-1)\ldots(n-k+1)$. We have

[TABLE]

By telescoping,

[TABLE]

Moreover,

[TABLE]

By our setting of parameters

[TABLE]

Therefore, for $j\in\{1,\ldots,r\}$,

[TABLE]

By Taylor series $2^{1/K}=\sum_{n=0}^{\infty}{\frac{(\ln 2)^{n}}{n!K^{n}}}>1+\frac{\ln 2}{K}>1+\frac{1}{4K}$, and thus $\frac{1}{1-(1+\frac{1}{4K})2^{-j/K}}\leq\frac{1}{1-2^{(1-j)/K}}$, for $j=2,\ldots,r$. For $j=1$, we have $\frac{1}{1-(1+\frac{1}{4K})2^{-\frac{1}{K}}}\leq 2^{K}$.

By Lemma 6, we have $\prod_{j=1}^{\infty}\frac{1}{1-2^{-j/K}}\leq 2^{5K}$. Therefore, the right hand side of (7) is upper bounded by $2^{6K}$. Together with (6), we prove (4) holds.

Finally, let $p={2^{6K}}/{{m\choose n_{r}}}$, we have $\operatorname*{\mathbb{P}}(S_{r}=T|X=x)\leq p$ and thus $H(S_{r}|X=x)\geq\log\frac{1}{p}=\log{{m\choose n_{r}}}-6K$. Therefore, $H(S_{r}|X)\geq\log{{m\choose n_{r}}}-6K$ and so $I(X;S_{r})=H(S_{r})-H(S_{r}|X)\leq 6K$. ∎

Lemma 6**.**

Let $K\in\mathbb{N}$ and $K\geq 1$. We have $\prod_{j=1}^{\infty}\frac{1}{1-2^{-j/K}}\leq 2^{5K}$.

Proof.

First, we bound the product of first $2K$ terms. Note that $\frac{1}{1-2^{-x}}\leq\frac{8}{3x}$ for $0<x\leq 2$. Therefore,

[TABLE]

Then, we bound the product of the rest terms

[TABLE]

Multiplying two parts proves the lemma. ∎

Theorem 2**.**

$\mathbf{R}^{\rightarrow,pub}_{\delta}(\mathbf{UR}^{\subset})=\Omega(\log\frac{1}{\delta}\log^{2}\frac{n}{\log(1/\delta)})$, given that $64\leq\log\frac{1}{\delta}\leq\frac{n}{64}$.

Proof.

By Lemma 3, the success probability of protocol $(\textsf{ENC},\textsf{DEC})$ is $1$. By Lemma 2, we have $\textsf{s}\geq\log(^{n}_{m})-\textsf{s}^{\prime}-1$, where $\textsf{s}^{\prime}=\log n+R+\operatorname*{\mathbb{E}}(\log(^{n}_{|B|}))$. The size of $B$ is $|B|=|S|-\sum_{r=1}^{R}{b_{r}}$. By Corollary 1, conditioned on $S$, $\operatorname*{\mathbb{P}}(b_{r}=0)\leq\frac{I(X;S_{r-1})+1}{\log\frac{1}{\delta}}$. By Lemma 5, $I(X;S_{r-1})\leq 6K$ (Note that when $r=1$, $I(X;S_{0})=0\leq 6K$). Therefore, $\operatorname*{\mathbb{E}}(b_{r})\geq 1-\frac{6K+1}{\log\frac{1}{\delta}}$. By the setting of parameters (see Algorithm 2) we have $\operatorname*{\mathbb{E}}(b_{r})\geq\frac{39}{64}$. Therefore, $\operatorname*{\mathbb{E}}(|B|)\leq|S|-\frac{39}{64}R$. By Lemma 4, $\log(^{n}_{m})-\operatorname*{\mathbb{E}}(\log(^{n}_{|B|}))\geq\frac{39}{64}R\cdot\log(\frac{n}{m}-1)\geq\frac{1}{2}R\log(\frac{n}{\log(1/\delta)})$. Furthermore, $\frac{1}{6}R\log\frac{n}{\log(1/\delta)}\geq R$. Thus we obtain $\textsf{s}\geq\frac{R}{3}\log\frac{n}{\log(1/\delta)}-(\log n+1)=\Omega(\log\frac{1}{\delta}\log^{2}\frac{n}{\log(1/\delta)})$. ∎

3.2 Communication Lower Bound for $\mathbf{UR}_{k}^{\subset}$

In this section, we prove the lower bound $\mathbf{R}^{\rightarrow,pub}_{1/2}(\mathbf{UR}^{\subset}_{k})=\Omega(\min\{n,k\log^{2}\frac{n}{k}\})$. In fact, our lower bound holds for any failure probability $\delta$ bounded away from $1$. Let $\mathcal{P}$ denote a $\mathbf{UR}_{k}^{\subset}$-protocol where Alice sends $\mathsf{Alice}_{k}(x)$ to Bob, and Bob outputs $\mathsf{Bob}_{k}(\mathsf{Alice}_{k}(x),y)$. We consider the following encoding/decoding scheme $(\textsf{ENC}_{k},\textsf{DEC}_{k})$ for $S\in{[n]\choose m}$. $\textsf{ENC}_{k}$ computes $M\leftarrow\mathsf{Alice}_{k}(\mathbf{1}_{S})$ as part of its message. In addition, $\textsf{ENC}_{k}$ includes $B\subseteq S$ constructed as follows, spending $\lceil\log{n\choose|B|}\rceil$ bits. Initially $B=S$, and $\textsf{ENC}_{k}$ proceeds in $R=\Theta(\log(n/k))$ rounds. Let $S_{0}=S\supseteq S_{1}\supseteq\ldots\supseteq S_{R}$ where $S_{r}$ is generated by sub-sampling each element in $S_{r-1}$ with probability $\frac{1}{2}$. In round $r$ ($r=1,\ldots,R$), $\textsf{ENC}_{k}$ tries to obtain $k$ elements from $S_{r-1}$ by invoking $\mathsf{Bob}_{k}(M,\mathbf{1}_{S\backslash S_{r-1}})$, denoted by $A_{k}$, and removes $A_{k}\cap(S_{r-1}\backslash S_{r})$ (whose expected size is $\frac{k}{2}$) from $B$. Note that $\textsf{DEC}_{k}$ is able to recover the elements in $A_{k}\cap(S_{r-1}\backslash S_{r})$. For each round the failure probability of $\mathsf{Bob}_{k}$ is at most $\delta$. Thus we have $\operatorname*{\mathbb{E}}(|S|-|B|)\geq\frac{k}{2}\cdot(1-\delta)\cdot R=\Omega(k\log\frac{n}{k})$. Furthermore, each element contains $\Theta(\log\frac{n}{k})$ bits of information, thus yielding a lower bound of $\Omega(k\log^{2}\frac{n}{k})$ bits.

In this section we assume $k\leq n/2^{10}$, since for larger $n$ we have an $\Omega(n)$ lower bound.

3.2.1 Encoding/decoding scheme

3.2.2 Analysis

Theorem 3**.**

$\mathbf{R}^{\rightarrow,pub}_{\delta}(\mathbf{UR}_{k}^{\subset})=\Omega((1-\delta)k\log^{2}\frac{n}{k})$, given that $1\leq k\leq\frac{n}{2^{10}}$ and $0<\delta\leq 1-\frac{50\log n}{k\log^{2}(n/k)}$.

Proof.

Let $S_{r}=S\cap T_{r}$. Let SUCC denote the event that $|S\cap T_{R}|=|S_{R}|\geq k$. Note that $\operatorname*{\mathbb{E}}|S_{R}|=\frac{1}{2^{R}}m=4k$. By the Chernoff bound, $\operatorname*{\mathbb{P}}(\textsf{SUCC})\geq\frac{1}{2}$. In the following, we argue conditioned on SUCC. Namely, in each round $r$, there are at least $k$ items in $S_{r}$.

Similar to Lemma 3, we can prove the protocol $(\textsf{ENC}_{k},\textsf{DEC}_{k})$ always succeeds. By Lemma 2, we have $\textsf{s}\geq\log(^{n}_{m})-\textsf{s}^{\prime}-2$, where $\textsf{s}^{\prime}=\log n+R+\operatorname*{\mathbb{E}}\log(^{n}_{|B|})$. The size of $B$ is $|B|=|S|-\sum_{r=1}^{R}{(b_{r}\cdot|A_{r}\cap(S_{r-1}\backslash S_{r})|)}$. The randomness used by $\mathcal{P}$ is independent from $S\backslash S_{r-1}$ for every $r\in[R]$. Therefore, $\operatorname*{\mathbb{E}}b_{r}\geq 1-\delta$, and $b_{r}$ is independent from $|A_{r}\cap(S_{r-1}\backslash S_{r})|$. We have $\operatorname*{\mathbb{E}}|A_{r}\cap(S_{r-1}\backslash S_{r})|=\frac{k}{2}$, and thus $\operatorname*{\mathbb{E}}(|S|-|B|)\geq\frac{(1-\delta)kR}{2}$. By Lemma 4, $\log(^{n}_{m})-\operatorname*{\mathbb{E}}\log(^{n}_{|B|})\geq\frac{(1-\delta)kR}{2}\cdot\log(\frac{n}{m}-1)\geq\frac{(1-\delta)kR}{5}\log(\frac{n}{k})$. Moreover, $R\leq\log n$ and $\log n\leq\frac{(1-\delta)kR}{12}\log\frac{n}{k}$. Thus we have $\textsf{s}=\Omega((1-\delta)kR\log\frac{n}{k})=\Omega((1-\delta)k\log^{2}\frac{n}{k})$. ∎

4 Lower bounds proofs via augmented indexing

Here we show another route to proving $\mathbf{R}^{\rightarrow,pub}_{\delta}(\mathbf{UR}_{k}^{\subset})=\Omega(\min\{n,t\log^{2}(n/t)\}$ via reduction from augmented indexing. We again separately prove lower bounds for $\mathbf{R}^{\rightarrow,pub}_{\delta}(\mathbf{UR}^{\subset})$ and $\mathbf{R}^{\rightarrow,pub}_{\frac{1}{5}}(\mathbf{UR}_{k}^{\subset})$. Both proofs make use of the following standard lemma. The proof can be found in the appendix (see Section A.2).

Lemma 7**.**

For any integers $u\geq 1$ and $1\leq m\leq u/(4e)$, there exists a collection $\mathcal{S}_{u,m}\subset\binom{[u]}{m}$ with $\log|\mathcal{S}_{u,m}|=\Theta(m\log(u/m))$ such that for all $S\neq S^{\prime}\in\mathcal{S}_{u,m}$, $|S\cap S^{\prime}|<m/2$.

Both our lower bounds in Sections 4.1 and 4.2 reduce from augmented indexing (henceforth $\mathbf{AugIndex}$) to either $\mathbf{UR}^{\subset}$ with low failure probability, or $\mathbf{UR}_{k}^{\subset}$ with constant failure probability, in the public coin one-way model of communication. We remind the reader of the setup for the $\mathbf{AugIndex}_{N}$ problem. There are two players, Charlie and Diane. Charlie receives $z\in\{0,1\}^{N}$ and Diane receives $j^{*}\in[N]$ together with $z_{j^{*}+1},\ldots,z_{N}$. Charlie must send a single message to Diane such that Diane can then output $z_{j^{*}}$. The following theorem is known.

Theorem 4**.**

[MNSW98]** $\mathbf{R}^{\rightarrow,pub}_{1/3}(\mathbf{AugIndex}_{N})=\Theta(N)$.

We show that if there is an $s$-bit communication protocol $\mathcal{P}$ for $\mathbf{UR}^{\subset}$ on $n$-bit vectors with failure probability $\delta$ (or for $\mathbf{UR}_{k}$ with constant failure probability), that implies the existence of an $s$-bit protocol $\mathcal{P}^{\prime}$ for $\mathbf{AugIndex}_{N}$ for some $N=\Theta(\log\frac{1}{\delta}\log^{2}\frac{n}{\log\frac{1}{\delta}})$ (or $N=\Theta(k\log^{2}(n/k))$ for $\mathbf{UR}_{k}$). The lower bound on $s$ then follows from Theorem 4.

4.1 Communication Lower Bound for $\mathbf{UR}^{\subset}$

Set $t=\log\frac{1}{\delta}$. In this section we assume $t<n/(4e)$ and show $\mathbf{R}^{\rightarrow,pub}_{\delta}(\mathbf{UR}^{\subset})=\Omega(t\log^{2}(n/t))$. This implies a lower bound of $\Omega(\min\{n,t\log^{2}(n/t)\})$ for all $\delta>0$ bounded away from $1$.

As mentioned, we assume we have an $s$-bit protocol $\mathcal{P}$ for $\mathbf{UR}^{\subset}$ with failure probability $\delta$, with players Alice and Bob.We use $\mathcal{P}$ to give an $s$-bit protocol $\mathcal{P}^{\prime}$ for $\mathbf{AugIndex}_{N}$, which has players Charlie and Diane, for $N=\Theta(t\log^{2}(n/t))$.

The protocol $\mathcal{P}^{\prime}$ operates as follows. Without loss of generality we may assume that, using the notation of Lemma 7, $|\mathcal{S}_{u,m}|$ is a power of $2$ for $u,m$ as in the lemma statement. This is accomplished by simply rounding $|\mathcal{S}_{u,m}|$ down to the nearest power of $2$ by removing elements arbitrarily. Also, define $L=c\log(n/t)$ for some sufficiently small constant $c\in(0,1)$ to be determined later. Now, Charlie partitions the bits of his input $z\in\{0,1\}^{N}$ into $L$ consecutive sequences of bits such that the $i$th chunk of bits for each $i\in[L]$ can be viewed as specifying an element $S_{i}\in\mathcal{S}_{u_{i},m}$ for $u_{i}=\frac{n}{100^{i}\cdot L}$ and $m=ct$. Lemma 7 gives $\log|\mathcal{S}_{u_{i},m}|=\Theta(m\log(u_{i}/m))$, which is $\Theta(t\log(n/t))$ for $c<1/14$. Thus $N=\Theta(L\cdot t\log(n/t))=\Theta(t\log^{2}(n/t))$. Given these sets $S_{1},\ldots,S_{L}$, we now discuss how Charlie generates a vector $x\in\{0,1\}^{n}$. Charlie then simulates Alice on $x$ to generate the message Alice would have send to Bob in protocol $\mathcal{P}$, then sends that same message to Diane.

To generate $x\in\{0,1\}^{n}$, assume Charlie and Diane have sampled a bijection from

[TABLE]

to $[n]$ uniformly at random. We denote this bijection by $\pi$. This is possible since $|A|=n$. Then Charlie defines $x$ to be the indicator vector $\mathbf{1}_{\pi(S)}$, where

[TABLE]

then sends a message $M$ to Diane, equal to Alice’s message with input $\mathbf{1}_{\pi(S)}$. This completes the description of Charlie’s behavior in the protocol $\mathcal{P}^{\prime}$.

We describe how Diane uses $M$ to solve $\mathbf{AugIndex}_{N}$. Diane’s input $j^{*}\in[N]$ lies in some chunk $i^{*}\in[L]$. We now show how Diane can use $\mathcal{P}$ to recover $S_{i^{*}}$ with probability $2/3$ (and thus in particular recover $z_{j^{*}}$). Since Diane knows $z_{j}$ for $j>j^{*}$, she knows $S_{i}$ for $i>i^{*}$. She can then execute the following algorithm.

In Algorithm 8 Diane is building up a subset $T_{i^{*}}$ of $S_{i^{*}}$. Once $|T_{i^{*}}|\geq|S_{i^{*}}|/2=m/2$, Diane can uniquely recover $S_{i^{*}}$ by the limited intersection property of $\mathcal{S}_{u_{i},m}$ guaranteed by Lemma 7. Until then, she uses $\mathcal{P}$ to recover elements of $S\backslash T$, which, as we now show, are chosen uniformly at random from $S\setminus T$.

Claim 4**.**

For every protocol for Alice and Bob that uses shared randomness with Bob’s behaviour given by $\mathsf{Bob}(\cdot)$, for every choice of shared random string $R$ of Alice and Bob, for every $S,T\subset S$, the following conditions hold. If $\pi$ is a uniformly random permutation, the success or failure of $\mathsf{Bob}(M,\mathbf{1}_{\pi(T)})$ is determined by $\{\pi(j)\}_{j\in T}$ and the image $\pi(S\setminus T)$ of $S\setminus T$ under $\pi$. Conditioned on a choice of $R$, $\{\pi(j)\}_{j\in T}$ and $\pi(S\setminus T)$ such that $\mathsf{Bob}(M,\mathbf{1}_{\pi(T)})$ succeeds, one has that $\pi^{-1}(\mathsf{Bob}(M,\mathbf{1}_{\pi(T)}))$ is a uniformly random element of $S\setminus T$.

Proof.

The first claim follows by noting that the message $M$ that Alice sends to Bob is solely a function of $R$ and $\pi(S)$. The behaviour of Bob is determined by $M$ and $\pi(T)$ (and the latter is determined by $\{\pi(j)\}_{j\in T}$).

Now condition on the values of $R$, $\{\pi(j)\}_{j\in T}$ and $\pi(S\setminus T)$ such that $\mathsf{Bob}(M,\mathbf{1}_{\pi(T)})$ succeeds, and let $j^{*}\in[n]$ denote the output. Note that by our conditioning $j^{*}$ is a fixed quantity. The only randomness left is the exact mapping of $S\setminus T$ to $\pi(S\setminus T)$. This mapping is independent of $\{\pi(j)\}_{j\in T}$ and $\pi(S\setminus T)$ and uniformly random, so $\pi^{-1}(j^{*})$ is a uniformly random element of $S\setminus T$, as required. ∎

Fix any protocol $\widetilde{\mathsf{Bob}}(M,\mathbf{1}_{\pi(T)})$ (not necessarily the one that Charlie and Diane use; see analysis of the idealized process $\widetilde{\mathcal{P}}$ below). Now fix $T$ together with values of $R$, $\{\pi(j)\}_{j\in T}$ and $\pi(S\setminus T)$ such that $\widetilde{\mathsf{Bob}}(M,\mathbf{1}_{\pi(T)})$ succeeds.

Elements in $S_{j},j<i^{*},$ are unlikely to be recovered.

Given Claim 4, since the elements of $S_{j}$ appear with frequency $100^{j}$ in $S\backslash T$, they are less likely to be returned by $\pi^{-1}(\widetilde{\mathsf{Bob}}(M,\mathbf{1}_{\pi(T)}))$ when $j$ is small. More precisely, as long as $|S_{i^{*}}\cap T_{i^{*}}|\geq m/2$, for any $1\leq j<i^{*}$

[TABLE]

Here again the probability is over the choice of $\pi|_{S\setminus T}:(S\setminus T)\to\pi(S\setminus T)$ (recall that we condition on the image $\pi(S\setminus T)$ under $\pi$, but not on the actual mapping).

We now define the set $\mathcal{T}$ of typical intermediate sets, which plays a crucial role in our analysis. Let $Q_{i}$ for $i\in[L]$ denote $\{i\}\times S_{i}\times[100^{i}]$. Let $\mathcal{T}$ be the collection of all $T\subset S$ such that (1) $Q_{i}\subset T$ for all $i>i^{*}$, and (2) for each $i<i^{*}$, $|T\cap Q_{i}|\leq 100^{i}\cdot m/4^{i^{*}-i}$. The following claim will be useful:

Claim 5**.**

For the set $\mathcal{T}$ defined above one has $|\mathcal{T}|=2^{O(m)}$.

Proof.

[TABLE]

∎

We will show that for most choices of $\pi$ and shared random string $R$ Algorithm 8 (a) never leaves the set $\mathcal{T}$ and (b) successfully terminates. Note that Algorithm 8 is a random process whose sample space is the product of the set of all possible permutations $\pi$ and shared random strings $R$. As before, we denote this process by $\mathcal{P}^{\prime}$. It is useful for analysis purposes to define another process $\widetilde{\mathcal{P}}$, which is an idealized version of $\mathcal{P}^{\prime}$. In this process instead of running $\mathsf{Bob}(M,\mathbf{1}_{\pi(T)})$ Alice runs $\widetilde{\mathsf{Bob}}(M,\mathbf{1}_{\pi(T)})$, which is guaranteed to output an element of $\pi(S\setminus T)$ for every choice of $T\subset S$, shared random string $R$, $\{\pi(j)\}_{j\in T}$, and $\pi(S\setminus T)$. The proof proceeds in three steps.

Step 1: proving that $\widetilde{\mathcal{P}}$ succeeds in recovering $T_{i^{*}}$ and never leaves $\mathcal{T}$ with high probability. Choose $\pi$ uniformly at random. By (11), as long as $|S_{i^{*}}\cap T_{i^{*}}|\geq m/2$, the expected number of items recovered by $\widetilde{\mathsf{Bob}}$ from $S_{i}$ for $i<i^{*}$ in the first $m$ iterations is at most $m/50^{i^{*}-i}$. Thus the probability of recovering more than $m/4^{i^{*}-i}$ items from $S_{i}$ is at most $(1/12)^{i^{*}-i}$ by Markov’s inequality. Note that the probability is over the choice of $\pi$ only, as $\widetilde{\mathsf{Bob}}$ is assumed to succeed with probability $1$ by definition of $\widetilde{\mathcal{P}}$. Thus

[TABLE]

In particular this means that with probability at least $1-1/10$ at most $\sum_{i<i^{*}}m/4^{i^{*}-i}<m/2$ items from $\bigcup_{i<i^{*}}S_{i}$ are recovered in the first $m$ (or fewer, if the algorithm terminates earlier) iterations. This also implies that with probability at least $1-1/10$ if the algorithm proceeds for the entire $m$ iterations, it recovers at least $m/2$ elements of $T_{i^{*}}$ and hence terminates. We thus get that $\widetilde{\mathcal{P}}$ succeeds at least with probability $1-1/10$.

Step 2: coupling $\widetilde{\mathcal{P}}$ to $\mathcal{P}^{\prime}$ on most of the probability space. For every $T\subset S$ and every $\pi$ let $\mathcal{E}_{T}(\pi)$ be the probabilistic event (over the choice of $\mathsf{Bob}$’s random string $R$) that $\mathsf{Bob}(M,\mathbf{1}_{\pi(T)})$ succeeds in returning an element in $\pi(S\backslash T)$. Note that $\mathcal{E}_{T}(\pi)$ is a subset of the probability space of shared random strings $R$, and depends on $\pi$. We let

[TABLE]

to simplify notation. Using Claim 5 and the union bound we have for every $\pi$

[TABLE]

as long as for $m=c\log(1/\delta)$ for $c$ a sufficiently small constant.

Now recall that $\widetilde{\mathsf{Bob}}(M,\mathbf{1}_{\pi(T)})$ is an idealized protocol, which is guaranteed to output an element of $\pi(S\setminus T)$ for every choice of $T\subset S$, shared random string $R$, $\{\pi(j)\}_{j\in T}$, and $\pi(S\setminus T)$. We have just shown that for every $\pi$ the event ${\mathcal{E}_{\mathcal{T}}(\pi)}$ occurs with probability at least $1-1/20$ over the choice of $R$. Now define $\widetilde{\mathsf{Bob}}(M,\mathbf{1}_{\pi(T)})$ as equal to $\mathsf{Bob}(M,\mathbf{1}_{\pi(T)})$ for all $T\in\mathcal{T}$ (the typical set of intermediate sets) and $(\pi,R)$ such that $R\in{\mathcal{E}_{\mathcal{T}}(\pi)}$, and extend $\widetilde{\mathsf{Bob}}(M,\mathbf{1}_{\pi(T)})$ to return an arbitrary element of $\pi(S\setminus T)$ for remaining tuples $(T,R,\pi(T),\pi(S\setminus T))$. Note that $\widetilde{\mathsf{Bob}}$ defined in this way is a deterministic function once $T$, $R$, $\pi(T)$ and $\pi(S\setminus T)$ are fixed.

Note that with probability at least $1-1/20$ over the choice of $\pi$ and $R$ one has $\mathsf{Bob}(M,\mathbf{1}_{\pi(T)})=\widetilde{\mathsf{Bob}}(M,\mathbf{1}_{\pi(T)})$ for all $T\in\mathcal{T}$, as required.

Step 3: arguing that $\mathcal{P}^{\prime}$ succeeds with high probability. Choose $(\pi,R)$ uniformly at random. By Step 2 we have that with probability at least $1-1/20$ over this choice $\mathsf{Bob}(M,\mathbf{1}_{\pi(T)})=\widetilde{\mathsf{Bob}}(M,\mathbf{1}_{\pi(T)})$ for all $T\in\mathcal{T}$. At the same time we have by Step 1 that with probability at least $1-1/10$ over the choice of $\pi$ the idealized process $\widetilde{\mathcal{P}}$ succeeds in recovering $T_{i^{*}}$ and never leaves $\mathcal{T}$. Putting the two bounds together, we get that $\mathcal{P}^{\prime}$ succeeds with probability at least $1-1/20-1/10>2/3$, showing the following theorem.

Theorem 5**.**

For any $0<\delta<1/2$ and integer $n\geq 1$ with $\log\frac{1}{\delta}<n/(4e)$, $\mathbf{R}^{\rightarrow,pub}_{\delta}(\mathbf{UR}^{\subset})\geq\mathbf{R}^{\rightarrow,pub}_{1/3}(\mathbf{AugIndex}_{N})$ for $N=\Theta(\log\frac{1}{\delta}\log^{2}\frac{n}{\log\frac{1}{\delta}})$.

Corollary 2**.**

For any $0<\delta<1/2$ and integer $n\geq 1$, $\mathbf{R}^{\rightarrow,pub}_{\delta}(\mathbf{UR}^{\subset})=\Omega(\min\{n,\log\frac{1}{\delta}\log^{2}\frac{n}{\log\frac{1}{\delta}}\})$.

4.2 Communication Lower Bound for $\mathbf{UR}_{k}^{\subset}$

The idea for lower bounding $\mathbf{R}^{\rightarrow,pub}_{\frac{1}{5}}(\mathbf{UR}_{k}^{\subset})$ is as in Section 4.1, but slightly simpler. That is because for the protocol $\mathcal{P}^{\prime}$ for $\mathbf{AugIndex}_{N}$, Diane will not make adaptive queries to Bob in the protocol $\mathcal{P}$ for $\mathbf{UR}_{k}^{\subset}$. Rather, she will only make one query using Bob and will be able to decide $\mathbf{AugIndex}_{N}$ with good probability from that single query. We make use of the following lemma from [JST11], whose proof is similar to our analysis in Section 4.1.

Lemma 8**.**

[JST11]** Any public coin protocol for $\mathbf{UR}^{\subset}$ can be turned into one that outputs every index $i\in[n]$ with $x_{i}\neq y_{i}$ with the same probability. The number of bits sent, failure probability, and number of rounds do not change. Similarly, any $\mathbf{UR}_{k}^{\subset}$ protocol can be turned into one in which all subsets of $[n]$ of size $\min\{k,\|x-y\|_{0}\}$ on which $x,y$ differ are equally likely to be output.

Henceforth we assume $\mathcal{P}$ outputs random differing indices, which is without loss of generality by Lemma 8.

Again Charlie receives $z\in\{0,1\}^{N}$ and Diane receives $j^{*}$ and $z_{j^{*}+1},\ldots,z_{N}$ and they want to solve $\mathbf{AugIndex}_{N}$. Charlie views his input as consisting of $L$ blocks for $L=c\log(n/k)$ for a sufficiently small constant $c\in(0,1)$, and the $i$th block for $i\in[L]$ specifies a set $S_{i}\in\mathcal{S}_{u_{i},m}$ for $m=ck$ and $u_{i}=n/(100^{i}L)$. As before, for $c$ sufficiently small we have $N=\Theta(L\cdot k\log(n/k))=\Theta(k\log^{2}(n/k))$. The bijection $A$ and set $S$ are defined exactly as in Section 4.1, and Charlie simulates Alice to send the message $M$ to Diane that Alice would have sent to Bob on input $\mathbf{1}_{S}$. Again, Diane knows $S_{i}$ for $i>i^{*}$, where $j^{*}$ lies in the $i^{*}$th block of bits. Diane’s algorithm to produce her output is then described in Algorithm 9.

Recall Bob, when he succeeds, returns $\min\{k,|S\backslash T|\}=k$ uniformly random elements from $S\backslash T$. Meanwhile, $S_{i^{*}}$ only has $m=ck$ elements for some small constant $c$. As in Section 4.1, almost all of the support of $S\backslash T$ comes from items in block $i^{*}$, and hence we expect almost all our $k$ samples to come from (and be uniform in) items corresponding to elements of $S_{i^{*}}$.

We now provide a formal analysis. Henceforth we condition on Bob succeeding, which happens with probability $4/5$. The number of elements in $S\backslash T$ corresponding to an element of $S_{i^{*}}$ is $100^{i^{*}}m$, whereas the number of elements corresponding to an element of $S_{i}$ for $i<i^{*}$ is

[TABLE]

Thus, we expect at most $k/99$ elements in $B$ to correspond to elements in $S_{i}$ for $i\neq i^{*}$, and the probability that we have at least $k/9$ such elements in $B$ is less than $1/10$ by Markov’s inequality. We henceforth condition on having less than $k/9$ such elements in $B$. Now we know $B$ contains at least $8k/9$ elements corresponding to $S_{i^{*}}$, chosen uniformly from $S_{i^{*}}\times[100^{i}]$. For any given element $a\in S_{i^{*}}$, the probability that none of the elements in $B$ from $S_{i^{*}}$ correspond to $a$ is $(1-1/m)^{\frac{8}{9}k}\leq e^{-(8/9)k/m}<1/30$ for $c$ sufficiently small (where $m=ck$). Thus the expected number of $a\in S_{i^{*}}$ not covered by $B$ is less than $m/30$. Thus the probability that fewer than $m/2$ elements are covered by $B$ is a most $1/15$ by Markov’s inequality (and otherwise, Diane succeeds). Thus, the probability that Diane succeeds is at least $4/5\cdot 9/10\cdot 14/15>2/3$. We have thus shown the following theorem.

Theorem 6**.**

For any integers $1\leq k\leq n$, $\mathbf{R}^{\rightarrow,pub}_{\frac{1}{5}}(\mathbf{UR}_{k}^{\subset})\geq\mathbf{R}^{\rightarrow,pub}_{\frac{1}{3}}(\mathbf{AugIndex}_{N})$ for $N=\Theta(k\log^{2}(n/k))$.

Corollary 3**.**

For any integers $1\leq k\leq n$, $\mathbf{R}^{\rightarrow,pub}_{\frac{1}{5}}(\mathbf{UR}_{k}^{\subset})=\Omega(\min\{n,k\log^{2}(n/k)\})$.

Remark 1**.**

One may wish to understand $\mathbf{R}^{\rightarrow,pub}_{\delta}(\mathbf{UR}_{k}^{\subset})$ for $\delta$ near $1$ (or at least, larger than $1/2$). Such a lower bound is given in Theorem 3. The proof given above as written would yield no lower bound in this regime for $\delta$ since $\mathbf{AugIndex}$ is in fact easy when the failure probability is allowed to be least $1/2$ (Charlie can send no message at all, and Diane can simply guess $z_{j^{*}}$ via a coin flip). One can however get a handle on $\mathbf{R}^{\rightarrow,pub}_{\delta}(\mathbf{UR}_{k}^{\subset})$ by instead directly reducing from the following variant of augmented indexing: Charlie receives $D\in\mathcal{S}_{u_{1},m}\times\cdots\times\mathcal{S}_{u_{L},m}$ and Diane receives $j^{*}\in[L]$ and $D_{j^{*}+1},\ldots,D_{L}$ and must output $D_{j^{*}}$, where the $u_{i}$ are as above. One can show that unless Charlie sends almost his entire input, Diane cannot have success probability significantly better than random guessing (which has success probability $O(\max_{i\in L}1/|\mathcal{S}_{u_{i},m}|)$). The proof is nearly identical to the analysis of augmented indexing over large domains [EJS10, JW13]. Indeed, the problem is even almost identical, except that here we consider Charlie receiving a vector whose entries come from different alphabet sizes (since the $|\mathcal{S}_{u_{i},m}|$ are different), whereas in [EJS10, JW13] all the entries come from the same alphabet. **

Acknowledgments

Initially the authors were focused on proving optimal lower bounds for samplers, but we thank Vasileios Nakos for pointing out that our $\mathbf{UR}^{\subset}$ lower bound immediately implies a tight lower bound for finding a duplicate in data streams as well. Also, initially our proof of Lemma 1 incurred an additive $1$ in the numerator of the right hand side of (2). This is clearly suboptimal for small $I(X;Y)$ (for example, consider $I(X;Y)=0$, in which case the right hand side should be $\delta$ and not $1/\log(1/\delta)$)). We thank T.S. Jayram for pointing out that a slight modification of our proof could actually replace the additive $1$ with the binary entropy function (and also for showing us a different proof of this lemma, which resembles the standard proof of Fano’s inequality).

Appendix A Appendix

A.1 A tight upper bound for $\mathbf{R}^{\rightarrow,pub}_{\delta}(\mathbf{UR}_{k})$

In [JST11, Proposition 1] it is shown that $\mathbf{R}^{\rightarrow,pub}_{\delta}(\mathbf{UR}_{k})=O(\min\{n,t\log^{2}n\})$ for $t=\max\{k,\log(1/\delta)\}$. Here we show that a minor modification of their protocol in fact shows the correct complexity $\mathbf{R}^{\rightarrow,pub}_{\delta}(\mathbf{UR}_{k})=O(\min\{n,t\log^{2}(n/t)\})$, which given our new lower bound, is optimal up to a constant factor for the full range of $n,k,\delta$ as long as $\delta$ is bounded away from $1$.

Recall Alice and Bob receive $x,y\in\{0,1\}^{n}$, respectively, and share a public random string. Alice must send a single message $M$ to Bob, from which Bob must recover $\min\{k,\|x-y\|_{0}\}$ indices $i\in[n]$ for which $x_{i}\neq y_{i}$. Bob is allowed to fail with probability $\delta$. The fact that $\mathbf{R}^{\rightarrow,pub}_{\delta}(\mathbf{UR}_{k})\leq n$ is obvious: Alice can simply send the message $M=x$, and Bob can then succeed with failure probability [math]. We thus now show $\mathbf{R}^{\rightarrow,pub}_{e^{-ck}}(\mathbf{UR}_{k})\leq k\log^{2}(n/k)$ for some constant $c>0$, which completes the proof of the upper bound. We assume $k\leq n/2$ (otherwise, Alice sends $x$ explicitly).

As mentioned, the protocol we describe is nearly identical to one in [JST11] (see also [CF14]). We will describe the new protocol here, then point out the two minor modifications that improve the $O(k\log^{2}n)$ bound to $O(k\log^{2}(n/k))$ in Remark 2. We first need the following lemma.

Lemma 9**.**

Let $\mathbb{F}_{q}$ be a finite field and $n>1$ an integer. Then for any $1\leq k\leq\frac{n}{2}$, there exists $\Pi_{k}\in\mathbb{F}_{q}^{m\times n}$ for $m=O(k\log_{q}(qn/k))$ s.t. for any $w\neq w^{\prime}\in\mathbb{F}_{q}^{n}$ with $\|w\|_{0},\|w^{\prime}\|_{0}\leq k$, $\Pi_{k}w\neq\Pi_{k}w^{\prime}$.

Proof.

The proof is via the probabilistic method. $\Pi_{k}w=\Pi_{k}w^{\prime}$ iff $\Pi_{k}(w-w^{\prime})=0$. Note $v=w-w^{\prime}$ has $\|v\|_{0}\leq 2k$. Thus it suffices to show that such a $\Pi_{k}$ exists with no $(2k)$-sparse vector in its kernel. The number of vectors $v\in\mathbb{F}_{q}^{n}$ with $\|v_{0}\|\leq 2k$ is at most $\binom{n}{2k}\cdot q^{2k}$. For any fixed $v$, $\operatorname*{\mathbb{P}}(\Pi_{k}v=0)=q^{-m}$. Thus

[TABLE]

by a union bound. The above is strictly less than $1$ for $m>2k+\log_{q}\binom{n}{2k}$, yielding the claim. ∎

Corollary 4**.**

Let $\mathbb{F}_{q}$ be a finite field and $n>1$ an integer. Then for any $1\leq k\leq\frac{n}{2}$, there exists $\Pi_{k}\in\mathbb{F}_{q}^{m\times n}$ for $m=O(k\log_{q}(qn/k))$ together with an algorithm $\mathcal{R}$ such that for any $w\in\mathbb{F}_{q}^{n}$ with $\|w\|_{0}\leq k$, $\mathcal{R}(\Pi_{k}w)=w$.

Proof.

Given Lemma 9, a simple such $\mathcal{R}$ is as follows. Given some $y=\Pi_{k}w^{*}$ with $\|w^{*}\|_{0}\leq k$, $\mathcal{R}$ loops over all $w$ in $\mathbb{F}_{q}^{n}$ with $\|w\|_{0}\leq k$ and outputs the first one it finds for which $\Pi_{k}w=y$. ∎

The protocol for $\mathbf{UR}_{k}$ is now as follows. Alice and Bob use public randomness to pick commonly known random functions $h_{0},\ldots,h_{L}:[n]\rightarrow\{0,1\}$ for $L=\lfloor\log_{2}(n/k)\rfloor$, such that for any $i\in[n]$ and for any $j$, $\operatorname*{\mathbb{P}}(h_{j}(i)=1)=2^{-j}$. They also agree on a matrix $\Pi_{16k}$ and $\mathcal{R}$ as described in Corollary 4 for a sufficiently large constant $C>0$ to be determined later, with $q=3$. Thus $\Pi_{16k}$ has $m=O(k\log(n/k))$ rows. Alice then computes $v_{j}=\Pi_{16k}x|_{h_{j}^{-1}(1)}$ for $j=0,\ldots,L$ where $v_{j}\in\mathbb{F}_{q}^{m}$, and her message to Bob is $M=(v_{0},\ldots,v_{L})$. For $S\subseteq[n]$ and $x$ an $n$-dimensional vector, $x|_{S}$ denotes the $n$-dimensional vector with $(x|_{S})_{i}=x_{i}$ for $i\in S$, and $(x|_{S})_{i}=0$ for $i\notin S$. Note Alice’s message $M$ is $O(k\log^{2}(n/k))$ bits, as desired. Bob then executes the following algorithm and outputs the returned values.

The correctness analysis is then as follows, which is nearly the same as the $\ell_{0}$-sampler of [JST11]. If Alice’s input is $x$ and Bob’s is $y$, let $a=x-y\in\{-1,0,1\}^{n}$, so that $a$ can be viewed as an element of $\mathbb{F}_{3}^{n}$. Also let $a_{j}=a|_{h_{j}^{-1}(1)}$. Then $\operatorname*{\mathbb{E}}\|v_{j}\|_{0}=\|a\|_{0}\cdot 2^{-j}$, and since $0\leq\|a\|_{0}\leq n$, there either (1) exists a unique $0\leq j^{*}\leq L$ such that $2k\leq\operatorname*{\mathbb{E}}\|a_{j}\|_{0}\cdot 2^{-j^{*}}<4k$, or (2) $\|a\|_{0}<2k$ (in which case we define $j^{*}=0$). Let $\mathcal{E}$ be the event that $\|a_{j}\|_{0}\leq 16k$ simultaneously for all $j\leq j^{*}$. Let $\mathcal{F}$ be the event that either we are in case (2), or we are in case (1) and $\|a_{j^{*}}\|_{0}\geq k$ holds. Note that conditioned on $\mathcal{E},\mathcal{F}$ both occurring, Bob succeeds by Corollary 4.

We now just need to show $\operatorname*{\mathbb{P}}(\neg\mathcal{E}\wedge\neg\mathcal{F})<e^{-\Omega(k)}$. We use the union bound. First, consider $\mathcal{F}$. If $j^{*}=0$, then $\operatorname*{\mathbb{P}}(\neg\mathcal{F})=0$. If $j^{*}\neq 0$, then $\operatorname*{\mathbb{P}}(\neg\mathcal{F})\leq\operatorname*{\mathbb{P}}(\|a_{j^{*}}\|_{0}<\frac{1}{2}\cdot\operatorname*{\mathbb{E}}\|a_{j^{*}}\|_{0})$, which is $e^{-\Omega(k)}$ by the Chernoff bound since $\operatorname*{\mathbb{E}}\|a_{j^{*}}\|_{0}=\Theta(k)$. Next we bound $\operatorname*{\mathbb{P}}(\neg\mathcal{E})$. For $j\geq j^{*}$, we know $\operatorname*{\mathbb{E}}\|a_{j}\|_{0}\leq 4k/2^{j-j^{*}}$. Thus, letting $\mu$ denote $\operatorname*{\mathbb{E}}\|a_{j}\|_{0}$,

[TABLE]

for some constant $C>0$ by the Chernoff bound and the fact that $16k/\mu\geq 4>e$. Recall that the Chernoff bound states that for $X$ a sum of independent Bernoullis,

[TABLE]

Then by a union bound over $j\geq j^{*}$ and applying (12),

[TABLE]

Remark 2**.**

As already mentioned, the protocol given above and the one described in [JST11] using $O(k\log^{2}n)$ bits differ in minor points. First: the protocol there used $\lfloor\log_{2}n\rfloor$ different hash functions $h_{j}$, but as seen above, only $\lfloor\log_{2}(n/k)\rfloor$ are needed. This already improves one $\log n$ factor to $\log(n/k)$. The other improvement comes from replacing the $k$-sparse recovery structure with $2k$ rows used in [JST11] with our Corollary 4. Note the matrix $\Pi_{k}$ in our corollary has even more rows, but the key point is that the bit complexity is improved. Whereas using a $k$-sparse recovery scheme as described in [JST11] would use $2k$ linear measurements of a $k$-sparse vector $w\in\{-1,0,1\}^{n}$ with $\log n$ bits per measurement (for a total of $O(k\log n)$ bits), we use $O(k\log(n/k))$ measurements with only $O(1)$ bits per measurement. The key insight is that we can work over $\mathbb{F}_{3}^{n}$ instead of $\mathbb{R}^{n}$ when the entries of $w$ are in $\{-1,0,1\}$, which leads to our slight improvement. **

A.2 Proof of the existence of the desired $\mathcal{S}_{u,m}$

Lemma 7 (restated). For any integers $u\geq 1$ and $1\leq m\leq u/(4e)$, there exists a collection $\mathcal{S}_{u,m}\subset\binom{[u]}{m}$ with $\log|\mathcal{S}_{u,m}|=\Theta(m\log(u/m))$ such that for all $S\neq S^{\prime}\in\mathcal{S}_{u,m}$, $|S\cap S^{\prime}|<m/2$.

Proof.

The proof is via the probabilistic method. We pick $S_{1},\ldots,S_{N}$ independently, each one uniformly at random from $\binom{[u]}{m}$. Fix $i\neq j\in[N]$. Imagine $S_{i}$ being fixed and picking the $m$ elements of $S_{j}$ one by one. Let $X_{k}$ denote the indicator random variable for the event that the $k$th element picked is also in $S_{i}$. Then $|S_{i}\cap S_{j}|=\sum_{k=1}^{m}X_{k}$, and we set $\mu:=\operatorname*{\mathbb{E}}|S_{i}\cap S_{j}|$, which is $m^{2}/u$ by linearity of expectation. We have $\operatorname*{\mathbb{P}}(|S_{i}\cap S_{j}|\geq m/2)=\operatorname*{\mathbb{P}}(|S_{i}\cap S_{j}|\geq(1+\delta)\mu)$ for $\delta=u/(2m)-1$. The $X_{k}$ are not independent, but they are negatively dependent. Thus the Chernoff bound yields

[TABLE]

Setting $N=\sqrt{(u/(2em))^{m/2}-1}$ so that ${N\choose 2}\leq N^{2}=(u/(2em))^{m/2}-1$, by a union bound with positive probability $|S_{i}\cap S_{j}|<m/2$ for all $i\neq j$, simultaneously, as desired. Note for this choice of $N$, we have $\log|\mathcal{S}_{u,m}|=\log N=\Theta(m\log(u/m))$. ∎