Crossing the Logarithmic Barrier for Dynamic Boolean Data Structure Lower Bounds

TL;DR

This paper establishes the first super-logarithmic lower bounds on the complexity of dynamic boolean data structures, introducing a novel method that combines communication protocols and polynomial techniques to prove these bounds.

Contribution

It introduces a new method for proving dynamic cell probe lower bounds, achieving the first super-logarithmic bounds for several fundamental problems.

Findings

Proves a ( ilde{\u2206} ext{log}^{1.5} n) lower bound for dynamic range counting.

Establishes ( ilde{\u2206} ext{log}^{1.5} n) lower bounds for boolean dynamic polynomial evaluation and 2D rectangle stabbing.

Provides the first ( ext{omega}( ext{log} n)) lower bounds for classical 2D range counting.

Abstract

This paper proves the first super-logarithmic lower bounds on the cell probe complexity of dynamic boolean (a.k.a. decision) data structure problems, a long-standing milestone in data structure lower bounds. We introduce a new method for proving dynamic cell probe lower bounds and use it to prove a $\tilde{\Omega}(\log^{1.5} n)$ lower bound on the operational time of a wide range of boolean data structure problems, most notably, on the query time of dynamic range counting over $\mathbb{F}_2$ ([Pat07]). Proving an $\omega(\lg n)$ lower bound for this problem was explicitly posed as one of five important open problems in the late Mihai P\v{a}tra\c{s}cu's obituary [Tho13]. This result also implies the first $\omega(\lg n)$ lower bound for the classical 2D range counting problem, one of the most fundamental data structure problems in computational geometry and spatial databases. We derive similar lower bounds for boolean versions of dynamic polynomial evaluation and 2D rectangle stabbing, and for the (non-boolean) problems of range selection and range median. Our technical centerpiece is a new way of "weakly" simulating dynamic data structures using efficient one-way communication protocols with small advantage over random guessing. This simulation involves a surprising excursion to low-degree (Chebychev) polynomials which may be of independent interest, and offers an entirely new algorithmic angle on the "cell sampling" method of Panigrahy et al. [PTW10].

Figures

Equations

$$t_q=\Omega \left(\min \left\{\frac{d\sqrt{\lg n}}{{\lg }^2(t_{u}w)},\frac{\sqrt{n}}{(t_{u}w{)}^{O(1)}}\right\}\right).$$

$$t_q=\Omega \left(\frac{{\lg }^{3/2}n}{{\lg }^3(t_{u}w)}\right).$$

$$\overrightarrow{\mathsf{adv}}(f,\mu ,m):=\sup \{ϵ\mid \text{ ∃ one-way protocol π that ϵ-solves f under μ with cost m}\}$$

$$\overrightarrow{\mathsf{adv}}\left(G_{\mathcal{P}}^i,\mathcal{U},n_i/(w\cdot t_u{)}^{a-1}\right)\ge \exp \left(-t_q{\lg }^2(w\cdot t_u)/\sqrt{\lg n}\right)$$

$$E[T_q^i]\le 8t_q/\ell \text{and}E[T_q]\le 8t_q,$$

$$\underset{\mathcal{U}}{\mathrm{Pr}}[T_q^i\le 32t_q/\ell ,T_q\le 32t_q]\ge 1/2.$$

$$p:=\frac{1}{{\left(w\cdot t_u\right)}^a}$$

$$\underset{C_0,\mathcal{U}}{\mathrm{Pr}}[{\mathcal{W}}_q,T_q\le 32t_q]\ge p^{32t_q/\ell }\cdot \underset{\mathcal{U}}{\mathrm{Pr}}[T_q^i\le 32t_q/\ell ,T_q\le 32t_q]\ge p^{32t_q/\ell }/2.$$

$${\mu }_q(z):=\underset{{\mathbf{U}}_i}{\mathrm{Pr}}[S_q\leftarrow z|u_{-i},c_0,c_2]$$

$${\eta }_q(z):=\underset{{\mathbf{U}}_i}{\mathrm{Pr}}[\mathcal{P}(q,\mathcal{U})=1,S_q\leftarrow z\mid u_{-i},c_0,c_2].$$

$$f(z)=f_{u_{-i},c_0,c_2}^q(z)$$

$$\underset{{\mathbf{U}}_i}{\mathrm{Pr}}\left[\mathcal{P}(q,\mathcal{U})=1\mid u_{-i},c_0,c_2,S_q{|}_Y\leftarrow y\right]>1/2.$$

$$O(p\cdot n_i\cdot t_{u}w)+O({\beta }^{i-1}\cdot t_{u}w)\le O(n_i/(w\cdot t_u{)}^{a-1}).$$

$$\frac{1}{2}$$

$$=\frac{1}{2}$$

$$|Y|=\kappa \le O\left(\sqrt{k\lg 1/ϵ}\right)\le O\left(t_q\lg (w\cdot t_u)/\sqrt{\lg n}\right),$$

$$\sum _{y\in [2^w{]}^Y}\left|\sum _{z\in [2^w{]}^{S_q}:z{|}_Y=y}{f}_{u_{-i},c_0,c_2}^q(z)\right|\ge \exp \left(-t_q\lg (w\cdot t_u)/\sqrt{\lg n}\right).$$

$$\underset{C_1}{\mathrm{Pr}}\left[Y\subseteq C_1\right]=p^{|Y|}=p^{O\left(t_q\lg (w\cdot t_u)/\sqrt{\lg n}\right)}=\exp \left(-t_q{\lg }^2(w\cdot t_u)/\sqrt{\lg n}\right).$$

$$1/2+\exp \left(-t_q\lg (w\cdot t_u)/\sqrt{\lg n}\right),$$

$$1/2+\exp \left(-t_q{\lg }^2(w\cdot t_u)/\sqrt{\lg n}\right).$$

$$\mathrm{Pr}[\mathcal{E}]\le 2\mathrm{Pr}\left[|C_0|\ge 2E\left[|C_0|\right]\right]\le \exp (-p(n_i\cdot t_u))\le \exp (-n_i/(w\cdot t_u{)}^a).$$

$$\underset{\mathcal{U},q}{\mathrm{Pr}}[\pi (q)\ne \mathcal{P}(q,\mathcal{U})]$$

$$\le 1/2-\exp \left(-t_q{\lg }^2(w\cdot t_u)/\sqrt{\lg n}\right)+\exp (-t_q\cdot (w\cdot t_u))$$

$$\le 1/2-\exp \left(-t_q{\lg }^2(w\cdot t_u)/\sqrt{\lg n}\right),$$

$$\frac{1}{|\mathcal{Q}|}\sum _{q\in Q}E[T_q^i]\le 2t_q/\ell ,$$

$$\overrightarrow{\mathsf{adv}}\left(G_{\mathcal{P}}^i,\mathcal{U},n_i/(w\cdot t_u{)}^{a-1}\right)\ge \exp \left(-t_q{\lg }^2(w\cdot t_u)/\sqrt{\lg n}\right)$$

$$\sum _{y\in {\Sigma }^Y}\left|\sum _{z{|}_Y=y}f(z)\right|\ge \exp \left(-\sqrt{k\cdot \lg 1/ϵ}\right).$$

Full text

We introduce a new method for proving dynamic cell probe lower bounds and use it to prove a $\tilde{\Omega}(\lg^{1.5}n)$ lower bound on the operational time of a wide range of boolean data structure problems, most notably, on the query time of dynamic range counting over $\mathbb{F}_{2}$ ([Pat07]). Proving an $\omega(\lg n)$ lower bound for this problem was explicitly posed as one of five important open problems in the late Mihai Pǎtraşcu’s obituary [Tho13]. This result also implies the first $\omega(\lg n)$ lower bound for the classical 2D range counting problem, one of the most fundamental data structure problems in computational geometry and spatial databases. We derive similar lower bounds for boolean versions of dynamic polynomial evaluation and 2D rectangle stabbing, and for the (non-boolean) problems of range selection and range median.

Our technical centerpiece is a new way of “weakly” simulating dynamic data structures using efficient one-way communication protocols with small advantage over random guessing. This simulation involves a surprising excursion to low-degree (Chebychev) polynomials which may be of independent interest, and offers an entirely new algorithmic angle on the “cell sampling” method of Panigrahy et al. [PTW10].

1 Introduction

Proving unconditional lower bounds on the operational time of data structures in the cell probe model [Yao81] is one of the holy grails of complexity theory, primarily because lower bounds in this model are oblivious to implementation considerations, hence they apply essentially to any imaginable data structure (and in particular, to the ubiquitous word-RAM model). Unfortunately, this abstraction makes it notoriously difficult to obtain data structure lower bounds, and progress over the past three decades has been very slow. In the dynamic cell probe model, where a data structure needs to maintain a database under an “online” sequence of $n$ operations (updates and queries) by accessing as few memory cells as possible, a number of lower bound techniques have been developed. In [FS89], Fredman and Saks proved $\Omega(\lg n/\lg\lg n)$ lower bounds for a list of dynamic problems. About 15 years later, Pǎtraşcu and Demaine [PD04, PD06] proved the first $\Omega(\lg n)$ lower bound ever shown for an explicit dynamic problem. The celebrated breakthrough work of Larsen [Lar12a] brought a near quadratic improvement on the lower bound frontier, where he showed an $\Omega((\lg n/\lg\lg n)^{2})$ cell probe lower bound for the 2D range sum problem (a.k.a. weighted orthogonal range counting in 2D). This is the highest cell probe lower bound known to date.

Larsen’s result has one substantial caveat, namely, it inherently requires the queries to have large ($\Theta(\lg n)$-bit) output size. Therefore, when measured per output-bit of a query, the highest lower bound remains only $\Omega(\lg n)$ per bit (for dynamic connectivity due to Pǎtraşcu and Demaine [PD06]).

In light of this, a concrete milestone that was identified en route to proving $\omega(\lg^{2}n)$ dynamic cell probe lower bounds, was to prove an $\omega(\lg n)$ cell probe lower bound for boolean (a.k.a. decision) data structure problems (the problem was explicitly posed in [Lar12a, Tho13, Lar13] and the caveat with previous techniques requiring large output has also been discussed in e.g. [Pat07, CGL15]). We stress that this challenge is provably a prerequisite for going beyond the $\omega(\lg^{2}n)$ barrier for general ($\Theta(\lg n)$-bit output) problems: Indeed, consider a dynamic data structure problem $\mathcal{P}$ maintaining a database with updates $\mathcal{U}$ and queries $\mathcal{Q}$, where each query $q\in\mathcal{Q}$ outputs $\lg n$ bits. If one could prove an $\omega(\lg^{2}n)$ lower bound for $\mathcal{P}$, this would directly translate into an $\omega(\lg n)$ lower bound for the following induced dynamic boolean problem $\mathcal{P}^{\mathrm{bool}}$: $\mathcal{P}^{\mathrm{bool}}$ has the same set of update operations $\mathcal{U}$, and has queries $\mathcal{Q}^{\prime}:=\mathcal{Q}\times[\lg n]$. Upon a query $(q,i)$, the data structure should output the $i$-th bit $(\mathcal{P}(q,\mathcal{U}))_{i}$ of the answer to the original query $q$ w.r.t the database $\mathcal{U}$. An $\omega(\lg n)$ lower bound then follows, simply because each query of $\mathcal{P}$ can be simulated by $\Theta(\lg n)$ queries of $\mathcal{P}^{\mathrm{bool}}$, and the update time is preserved. Thus, to break the $\lg^{2}n$-barrier for cell probe lower bounds, one must first prove a super-logarithmic lower bound for some dynamic boolean problem. Of course, many classic data structure problems are naturally boolean (e.g., reachability, membership, etc.), hence studying decision data structure problems is interesting on its own.

Technically speaking, the common reason why all previous techniques hitherto (e.g., [Pat07, Lar12a, WY16]) fail to prove super-logarithmic lower bounds for dynamic boolean problems, is that they all heavily rely on each query revealing a large amount of information about the database. In contrast, for boolean problems, each query could reveal at most one bit of information, and thus any such technique is doomed to fail. We elaborate on this excruciating obstacle and how we overcome it in the following subsection.

In this paper, we develop a fundamentally new lower bound method and use it to prove the first super-logarithmic lower bounds for dynamic boolean data structure problems. Our results apply to natural boolean versions of several classic data structure problems. Most notably, we study a boolean variant of the dynamic 2D range counting problem. In 2D range counting, $n$ points are inserted one-by-one into an $[n]\times[n]$ integer grid, and given a query point $q=(x,y)\in[n]\times[n]$, the data structure must return the number of points $p$ dominated by $q$ (i.e., $p.x\leq x$ and $p.y\leq y$). This is one of the most fundamental data structure problems in computational geometry and spatial database theory (see e.g., [Aga04] and references therein). It is known that a variant of dynamic “range trees” solve this problem using $O((\lg n/\lg\lg n)^{2})$ amortized update time and $O((\lg n/\lg\lg n)^{2})$ worst case query time ([BGJS11]). We prove an $\tilde{\Omega}(\lg^{1.5}n)$ lower bound even for a boolean version, called 2D range parity, where one needs only to return the parity of the number of points dominated by $q$. This is, in particular, the first $\omega(\lg n)$ lower bound for the (classical) 2D range counting problem. We are also pleased to report that this is the first progress made on the 5 important open problems posed in Mihai Pǎtraşcu’s obituary [Tho13].

In addition to the new results for 2D range parity, we also prove the first $\omega(\lg n)$ lower bounds for the classic (non-boolean) problems of dynamic range selection and range median, as well as an $\omega(\lg n)$ lower bound for a boolean version of polynomial evaluation. We formally state these problems, our new lower bounds, and a discussion of previous state-of-the-art bounds in Section 1.2.

The following two subsections provide a streamlined overview of our technical approach and how we apply it to obtain new dynamic lower bounds, as well as discussion and comparison to previous related work.

1.1 Techniques

To better understand the challenge involved in proving super-logarithmic lower bounds for boolean data structure problems, and how our approach departs from previous techniques that fail to overcome it, we first revisit Larsen’s $\tilde{\Omega}(\lg^{2}n)$ lower bound technique for problems with $\Theta(\lg n)$-bit output size, which is most relevant for our work. (We caution that a few variations [CGL15, WY16] of Larsen’s [Lar12a] approach have been proposed, yet all of them crucially rely on large query output size). The following overview is presented in the context of the 2D range sum problem for which Larsen originally proved his lower bound. 2D range sum is the variant of 2D range counting where each point is assigned a $\Theta(\lg n)$-bit integer weight, and the goal is to return the sum of weights assigned to points dominated by the query $q$. Clearly this is a harder problem than 2D range counting (which corresponds to all weights being $1$) and 2D range parity (which again has all weights being $1$, but now only $1$ bit of the output must be returned).

Larsen’s Lower Bound [Lar12a].

Larsen’s result combines the seminal chronogram method of Fredman and Saks [FS89] together with the cell sampling technique introduced by Panigrahy et al. [PTW10]. The idea is to show that, after $n$ random updates have been performed,111Each update inserts a random point and assigns it a random $\Theta(\lg n)$-bit weight. any data structure (with $\mathrm{poly}\lg n$ update time) must probe many cells when prompted on a random range query. To this end, the $n$ random updates are partitioned into $\ell:=\Theta(\lg n/\lg\lg n)$ epochs $\mathbf{U}_{\ell},\ldots,\mathbf{U}_{i},\ldots,\mathbf{U}_{1}$, where the $i$-th epoch $\mathbf{U}_{i}$ consists of $\beta^{i}$ updates for $\beta=\mathrm{poly}\lg n$. The goal is to show that, for each epoch $i\in\{1,\ldots,\ell\}$, a random query must read in expectation $\Omega(\lg n/\lg\lg n)$ memory cells whose last modification occurred during the $i$th epoch $\mathbf{U}_{i}$. Summing over all epochs then yields a $\tilde{\Omega}(\lg^{2}n)$ query lower bound.

To carry out this approach, one restricts the attention to epoch $i$, assuming all remaining updates in other epochs ($\mathbf{U}_{-i}$) are fixed (i.e., only $\mathbf{U}_{i}$ is random). For a data structure $D$, let $A_{i}$ denote the set of memory cells associated with epoch $i$, i.e., the cells whose last update occurred in epoch $i$. Clearly, any cell that is written before epoch $i$ cannot contain any information about $\mathbf{U}_{i}$, while the construction guarantees there are relatively few cells written after epoch $i$, due to the geometric decay in the lengths of epochs. Thus, “most” of the information $D$ provides on $\mathbf{U}_{i}$ comes from cell probes to $A_{i}$ (hence, intuitively, the chronogram method reduces a dynamic problem into $\approx\lg n$ nearly independent static problems).

The high-level idea is to now prove that, if a too-good-to-be-true data structure $D$ exists, which probes $o(\lg n/\lg\lg n)$ cells associated with epoch $i$ on an average query, then $D$ can be used to devise a compression scheme (i.e., a “one-way” communication protocol) which allows a decoder to reconstruct the random update sequence $\mathbf{U}_{i}$ from an $o(H(\mathbf{U}_{i}))$-bit message, an information-theoretic contradiction.

Larsen’s encoding scheme has the encoder (Alice) find a subset $C\subseteq A_{i}$ of a fixed size, such that sufficiently many range queries $q\in[n]\times[n]$ can be resolved by $C$, meaning that these queries can be answered without probing any cell in $A_{i}\setminus C$. Indeed, the assumption that the query algorithm of $D$ probes only $o(\lg n/\lg\lg n)$ cells from $A_{i}$, implies that a random subset of size $|C|=|A_{i}|/\mathrm{poly}\lg n$ cells resolves at least a $(1/\mathrm{poly}\lg n)^{o(\lg n/\lg\lg n)}=n^{-o(1)}$-fraction of the $n^{2}$ possible queries, an observation first made in [PTW10]. This observation in turn implies that by sending the contents and addresses of $C$, the decoder (Bob) can recover the answers to some specific subset $Q^{*}\subseteq[n]\times[n]$ of at least $n^{2-o(1)}$ queries. Intuitively, if the queries of the problem are “sufficiently independent”, e.g., the answers to all queries are $n$-wise independent over a random $\mathbf{U}_{i}$, then answering $Q^{*}$ or even any subset of $Q^{*}$ of size $n$ would be sufficient to reconstruct the entire update sequence $\mathbf{U}_{i}$. Thus, by simulating the query algorithm $\forall q\in Q^{*}$ and using the set $C$ to “fill in” his missing memory cells associated with $\mathbf{U}_{i}$, Bob could essentially recover $\mathbf{U}_{i}$. On the other hand, the update sequence itself contains at least $\Omega(|\mathbf{U}_{i}|)\gg|C|\cdot w$ bits of entropy, hence it cannot possibly be reconstructed from $C$, yielding an information-theoretic contradiction. Here, and throughout the paper, $w$ denotes the number of bits in a memory cell. We make the standard assumption that $w=\Omega(\lg n)$, such that a cell has enough bits to store an index into the sequence of updates performed.

It is noteworthy that range queries do not directly possess such “$n$-wise independence” property per-se, but using (nontrivial) technical manipulations (a-la [Pat07, Lar12a, WY16]) this argument can be made to work, see the discussion in Section 6.

Alas, a subtle but crucial issue with the above scheme is that Bob cannot identify the subset $Q^{*}$, that is, when simulating the query algorithm of $D$ on a given query, he can never know whether an unsampled ($\notin C$) encountered cell in the query-path in fact belongs to $A_{i}$ or not. This issue is also faced by Pǎtraşcu’s approach in [Pat07]. Larsen resolves this excruciating problem by having Alice further send Bob the indices of (a subset of) $Q^{*}$ that already reveals enough information about $\mathbf{U}_{i}$ to get a contradiction. In order to achieve the anticipated contradiction, the problem must therefore guarantee that the answer to a query reveals more information than it takes to specify the query itself ($\Theta(\lg n)$ bits for 2D range sum). This is precisely the reason why Larsen’s lower bound requires $\Omega(\lg n)$-bit weights assigned to each input point, whereas for the boolean 2D range parity problem, all bets are off.

1.1.1 Our Techniques

We develop a new lower bound technique which ultimately circumvents the aforementioned obstacle that stems from Bob’s inability to identify the subset $Q^{*}$. Our high-level strategy is to argue that an efficient dynamic data structure for a boolean problem, induces an efficient one-way protocol from Alice (holding the entire update sequence $\mathcal{U}:=\mathbf{U}_{\ell},\ldots,\mathbf{U}_{1}$ as before) to Bob (who now receives a query $q\in\mathcal{Q}$ and $\mathcal{U}\setminus\{\mathbf{U}_{i}\}$), which enables Bob to answer his boolean query with some tiny yet nontrivial advantage over random guessing. For a dynamic boolean data structure problem $\mathcal{P}$, we denote this induced communication game (corresponding to the $i$th epoch) by $G^{i}_{\mathcal{P}}$. The following “weak simulation” theorem, which is the centerpiece of this paper, applies to any dynamic boolean data structure problem $\mathcal{P}$:

Theorem 1** (One-Way Weak Simulation Theorem, informal).**

Let $\mathcal{P}$ be any dynamic boolean data structure problem, with $n$ random updates grouped into epochs $\;\mathcal{U}=\{\mathbf{U}_{i}\}_{i=1}^{\ell}$ followed by a single query $q\in\mathcal{Q}$. If $\mathcal{P}$ admits a dynamic data structure $D$ with word-size $w$, worst-case update time $t_{u}$ and average (over $\mathcal{Q}$) expected query time $t_{q}$ with respect to $\mathcal{U}$, satisfying $t_{q},t_{u},w\leq n^{0.1}$, then there exists some epoch $i\in[\ell]$ for which there is a one-way randomized communication protocol for $G^{i}_{\mathcal{P}}$ in which Alice sends Bob a message of only $|\mathbf{U}_{i}|/(wt_{u})^{\Theta(1)}$ bits, and after which Bob successfully computes $\mathcal{P}(q,\mathcal{U})$ with probability at least $1/2+\exp\left(-t_{q}\lg^{2}(w\cdot t_{u})/\sqrt{\lg n}\right)$.222Throughout the paper, we use $\exp(x)$ to denote $2^{\Theta(x)}$.

The formal statement and proof of the above theorem can be found in Section 4. Before we elaborate on the proof of Theorem 1, let us explain informally why such a seemingly modest guarantee suffices to prove super-logarithmic cell probe lower bounds on boolean problems with a certain “list-decoding” property. If we view query-answering as mapping an update sequence to an answer vector,333An answer vector is a $|\mathcal{Q}|$-dimensional vector containing one coordinate per query, whose value is the answer to this query. then answering a random query correctly with probability $1/2+e^{-r(n)}$ would correspond to mapping an update sequence to an answer vector that is $(1/2-e^{-r(n)})$-far from the true answer vector defined by the problem. Intuitively, if the correct mapping defined by the problem is list-decodable in the sense that in the $(1/2-e^{-r(n)})$-ball centered at any answer vector, there are very few codewords (which are the correct answer vectors corresponding to some update sequences), then knowing any vector within distance $(1/2-e^{-r(n)})$ from the correct answer vector would reveal a lot of information about the update sequence. Standard probabilistic arguments [Vad12] show that when the code rate is $n^{-\Theta(1)}$ (i.e., $|\mathcal{Q}|=n^{\Theta(1)}$ as for 2D range parity), a random code is “sufficiently” list-decodable with $r(n)=\Omega(\lg n)$, i.e., for most data structure problems, the protocol in the theorem would reveal too much information if Bob can predict the answer with probability, say $1/2+e^{-0.01\lg n}$. Therefore, Theorem 1 would imply that the query time must be at least $t_{q}=\Omega(\frac{\lg^{1.5}n}{\lg^{2}(w\cdot t_{u})})$. Assuming the data structure has $t_{u}=\mathrm{poly}\lg n$ worst-case update time and standard word-size $w=\Theta(\lg n)$, the above bound gives $t_{q}\geq\tilde{\Omega}(\lg^{1.5}n)$. Indeed all our concrete lower bounds are obtained by showing a similar list-decoding property with $r(n)=\Omega(\lg n)$, yielding a lower bound of $\tilde{\Omega}(\lg^{1.5}n)$. See Subsection 1.2 for more details.

Overview of Theorem 1 and the “Peak-to-Average” Lemma.

We now present a streamlined overview of the technical approach and proof of our weak one-way simulation theorem, the main result of this paper. Let $\mathcal{P}$ be any boolean dynamic data structure problem and denote by $n_{i}:=|\mathbf{U}_{i}|=\beta^{i}$ the size of each epoch of random updates (where $\beta:=(t_{u}\cdot w)^{\Theta(1)}$ and $\sum_{i=1}^{\ell}n_{i}=n$). Recall that in $G^{i}_{\mathcal{P}}$, Alice receives the entire sequence of epochs $\mathcal{U}$, Bob receives $q\in_{R}\mathcal{Q}$ and $\mathcal{U}\setminus\{\mathbf{U}_{i}\}$, and our objective is to show that Alice can send Bob a relatively short message ($n_{i}/(t_{u}\cdot w)^{\Theta(1)}$ bits) which allows him to compute the answer to $q$ w.r.t $\mathcal{U}$, denoted $\mathcal{P}(q,\mathcal{U})\in\{0,1\}$, with advantage $\delta:=\exp(-t_{q}\lg^{2}(w\cdot t_{u})/\sqrt{\lg n})$ over $1/2$.

Suppose $\mathcal{P}$ admits a dynamic data structure $D$ with worst-case update time $t_{u}$ and expected query time $t_{q}$ with respect to $\mathcal{U}$ and $q\in_{R}\mathcal{Q}$. Following Larsen’s cell sampling approach, a natural course of action for Alice is to generate the updated memory state $M$ of $D$ (w.r.t $\mathcal{U}$), and send Bob a relatively small random subset $C_{0}$ of the the cells $A_{i}$ associated with epoch $i$, where each cell is sampled with probability $p=1/(t_{u}\cdot w)^{\Theta(1)}$. Since the expected query time of $D$ is $t_{q}$ and there are $\ell=\Theta(\lg_{\beta}n)$ epochs, the average (over $i\in[\ell]$) number of cells in $A_{i}$ probed by a query is $t_{q}/\ell$, hence the probability that Alice’s random set $C_{0}$ resolves Bob’s random query $q\in_{R}\mathcal{Q}$ is at least $\epsilon:=p^{\Theta(t_{q}/\ell)}$. Let us henceforth denote this desirable event by $\mathcal{W}_{q}$. It is easy to see that, if Alice further sends Bob all cells that were written (associated) with future epochs $\mathbf{U}_{<i}$ (which can be done using less than $n_{i}/(w\cdot t_{u})^{\Theta(1)}$ bits due to the geometric decay of epochs and the assumption that $D$ probes at most $t_{u}$ cells on each update operation), then conditioned on $\mathcal{W}_{q}$, Bob would have acquired all the necessary information to perfectly simulate the correct query-path of $D$ on his query $q$.

Thus, if Bob could detect the event $\mathcal{W}_{q}$, the above argument would have already yielded an advantage of roughly $\Pr[\mathcal{W}_{q}]\geq\epsilon=p^{\Theta(t_{q}/\ell)}\geq\exp(-t_{q}\lg^{2}(w\cdot t_{u})/\lg n)\gg\delta$ (as Bob could simply output a random coin-toss unless $\mathcal{W}_{q}$ occurs), and this would have finished the proof. Unfortunately, certifying the occurrence of $\mathcal{W}_{q}$ is prohibitively expensive, precisely for the same reason that identifying the subset $Q^{*}$ is costly in Larsen’s argument. Abandoning the hope for certifying the event $\mathcal{W}_{q}$ (while insisting on low communication) means that we must take a fundamentally different approach to argue that the noticeable occurrence of this event can somehow still be exploited implicitly so as to guarantee a nontrivial advantage. This is the heart of the paper, and the focal point of the rest of this exposition.

The most general strategy Bob has is to output his “maximum likelihood” estimate for the answer $\mathcal{P}(q,\mathcal{U})$ given the information he receives, i.e., the more likely posterior value of $(\mathcal{P}(q,\mathcal{U})\;|\mathbf{U}_{-i},C_{0})\in\{0,1\}$ (for simplicity of exposition, we henceforth ignore the conditioning on $\mathbf{U}_{-i},C_{0}$ and on the set of updates $D$ makes to future epochs $\mathbf{U}_{<i}$ which Alice sends as well). Assuming without loss of generality that the answer to the query is $\mathcal{P}(q,\mathcal{U})=1$, when $\mathcal{W}_{q}$ occurs, this strategy produces an advantage (“bias”) of $1/2$ (since when $\mathcal{W}_{q}$ occurs, the answer $\mathcal{P}(q,\mathcal{U})$ is completely determined by $\mathbf{U}_{-i},C_{0}$ and the updates to $\mathbf{U}_{<i}$), and when it does not occur, the strategy produces a bias of $\Pr[(\mathcal{P}(q,\mathcal{U})=1|\overline{\mathcal{W}_{q}})]-1/2$. Thus, the overall bias is $\Pr[\mathcal{W}_{q}]\cdot(1/2)+\Pr[\overline{\mathcal{W}_{q}}]\cdot\left(\Pr[(\mathcal{P}(q,\mathcal{U})=1|\overline{\mathcal{W}_{q}})]-1/2\right)$. This quantity could be arbitrarily close to [math], since we have no control over the distribution of the answer conditioned on the complement event $\overline{\mathcal{W}_{q}}$, which might even cause perfect cancellation of the two terms.

Nevertheless, one could hope that such unfortunate cancellation of our advantage can be avoided if Alice reveals to Bob some little extra “relevant” information. To be more precise, let $S_{q}$ be the set of memory addresses $D$ would have probed when invoked on the query $q$ according to Bob’s simulation. That is, Bob simulates $D$ until epoch $i$, updates the contents for all cells that appear in Alice’s message, and simulates the query algorithm for $q$ on this memory state. In particular, if the event $\mathcal{W}_{q}$ occurs, then $S_{q}$ is the correct set of memory cells the data structure probes. Of course, the set $S_{q}$ is extremely unlikely to be “correct” as $\Pr[\mathcal{W}_{q}]$ is tiny, so $S_{q}$ should generally be viewed as an arbitrary subset of memory addresses. Now, the true contents of the cells $S_{q}$ (w.r.t the true memory state $M$) induce some posterior distribution on the correct answer $\mathcal{P}(q,\mathcal{U})$ (in particular, when $\mathcal{W}_{q}$ occurs, the path is correct and its contents induce the true answer).

Imagine that Alice further reveals to Bob the true contents of some small subset $Y\subseteq S_{q}$, i.e., an assignment $x\in[2^{w}]^{Y}$. The posterior distribution of the answer $\mathcal{P}(q,\mathcal{U})$ conditioned on $x$ is simply the convex combination of the posterior distributions conditioned on “$S_{q}=z$” for all $z$’s that are consistent with $x$ ($z|_{Y}=x$), weighted by the probability of $z$ ($\Pr[S_{q}=z]$) up to some normalizer. The contribution of each term in this convex combination (i.e., of each posterior distribution induced by a partial assignment $x$) to the overall bias, is precisely the average, over all full assignments $z$ to cells in $S_{q}$ which are $x$-consistent, of the posterior bias induced by the event “$S_{q}=z$” (i.e., when the entire $S_{q}$ is revealed). For each full assignment $z$, we denote its latter contribution by $f(z)$, hence the expected bias contributed by the event “$z|_{Y}=x$” is nothing but the sum of $f(z)$ over all $z$’s satisfying $z|_{Y}=x$. Furthermore, we know that there is some assignment $z^{*}$, namely the contents of $S_{q}$ when $\mathcal{W}_{q}$ occurs, such that $|f(z^{*})|$ is “large” (recall the bias is $1/2$ in this event). Thus, the key question we pose and set out to answer, is whether it is possible to translate this $\ell_{\infty}$ “peak” of $f$ into a comparable lower bound on the “average” bias $\sum_{x}\left|\sum_{z|_{Y}=x}f(z)\right|$, by conditioning on the assignments to a small subset of coordinates $Y$. Indeed, if such $Y$ exists, Alice can sample independently another set of memory cells $C_{1}$ and send it to Bob. With probability $p^{|Y|}$, all contents of $Y$ are revealed to Bob, and we will have the desired advantage. In essence, the above question is equivalent to the following information-theoretic problem:

Let $Z$ be a $k$-variate random variable and $B$ a uniform binary random variable in the same probability space, satisfying: (i) $\Pr[Z=z^{*}]\geq\epsilon$ for some $z^{*}$;   (ii) $H(B\mid Z=z^{*})=0$. What is the smallest subset of coordinates $Y\subseteq[k]$ such that $H(B\mid(Z|_{Y}))\leq 1-\eta$ ? *

The crux of our proof is the following lemma, which asserts that conditioning on only $|Y|=O(\sqrt{k\lg(1/\epsilon)})$ many coordinates suffices to achieve a non-negligible average advantage $\eta=\exp(-\sqrt{k\lg(1/\epsilon)})$.

Lemma 1** (Peak-to-Average Lemma).**

Let $f:\Sigma^{k}\to\mathbb{R}$ be any real function satisfying: (i) $\sum_{z\in\Sigma^{k}}|f(z)|\leq 1$; and (ii) $\max_{z\in\Sigma^{k}}|f(z)|\geq\epsilon$. Then there exists a subset $Y$ of indices, $|Y|\leq O\left(\sqrt{k\cdot\lg 1/\epsilon}\right)$, such that $\sum_{y\in\Sigma^{Y}}|\sum_{z|_{Y}=y}f(z)|\geq\exp(-\sqrt{k\cdot\lg 1/\epsilon})$.

An indispensable ingredient of the proof is the usage of low-degree (multivariate) polynomials with “threshold”-like phenomena, commonly known as (discrete) Chebyshev polynomials.444These are real polynomials defined on the $k$-hypercube, of degree $O(\sqrt{k\lg(1/\gamma)})$ and whose value is uniformly bounded by $\gamma$ everywhere on the cube except the all-[math] point which attains the value $1$. The lemma can be viewed as an interesting and efficient way of “decomposing” a distribution into a small number of conditional distributions, “boosting” the effect of a single desirable event, hence the Peak-to-Average Lemma may be of independent interest (see Section 4.1 for a high-level overview and the formal proof). In Appendix B, we show that the lemma is in fact tight, in the sense that there are functions for which conditioning on $o\left(\sqrt{k\cdot\lg 1/\epsilon}\right)$ of their coordinates provides no advantage at all.

To complete the proof of the simulation theorem, we apply the Peak-to-Average Lemma with $f$, $k:=t_{q}$ and $\epsilon:=p^{\Theta(t_{q}/\ell)}=(1/wt_{u})^{O(t_{q}/\ell)}$. The lemma guarantees that Bob can find a small (specific) set of coordinates $Y\subseteq S_{q}$, such that his maximum-likelihood estimate conditioned on the true value $y$ of the coordinates in $Y$ must provide an advantage of at least $\exp(-\sqrt{k\cdot\lg 1/\epsilon})=\exp\left(-t_{q}\lg(w\cdot t_{u})/\sqrt{\lg n}\right)$. Since $|Y|$ is small, the probability that $Y$ is contained in Alice’s second sample $C_{1}$ is $p^{|Y|}\geq\exp\left(-t_{q}\lg^{2}(w\cdot t_{u})/\sqrt{\lg n}\right)$. Overall, Bob’s maximum-likelihood strategy provides the desired advantage $\delta$ we sought.

1.2 Applications: New Lower Bounds

We apply our new technique to a number of classic data structure problems, resulting in a range of new lower bounds. This section describes the problems and the lower bounds we derive for them, in context of prior work. As a warm-up, we prove a lower bound for a somewhat artificial version of polynomial evaluation:

Polynomial Evaluation.

Consider storing, updating and evaluating a polynomial $P$ over the Galois field $GF(2^{d})$. Here we assume that elements of $GF(2^{d})$ are represented by bit strings in $\{0,1\}^{d}$, i.e. there is some bijection between $GF(2^{d})$ and $\{0,1\}^{d}$. Elements are represented by the corresponding bit strings. Any bijection between elements and bit strings suffice for our lower bound to apply.

The least-bit polynomial evaluation data structure problem is defined as follows: A degree $n\leq 2^{d/4}$ polynomial $P(x)=\sum_{i=0}^{n}a_{i}x^{i}$ over $GF(2^{d})$ is initialized with all $n+1$ coefficients $a_{i}$ being [math]. An update is specified by a tuple $(i,b)$ where $i\in[n+1]$ is an index and $b$ is an element in $GF(2^{d})$. It changes the coefficient $a_{i}$ such that $a_{i}\leftarrow a_{i}+b$ (where addition is over $GF(2^{d})$). A query is specified by an element $y\in GF(2^{d})$ and one must return the least significant bit of $P(y)$. Recall that we make no assumptions on the concrete representation of the elements in $GF(2^{d})$, only that the elements are in a bijection with $\{0,1\}^{d}$ so that precisely half of all elements in $GF(2^{d})$ have a [math] as the least significant bit.

Using our weak one-way simulation theorem, Section 5 proves the following lower bound:

Theorem 2**.**

Any cell probe data structure for least-bit polynomial evaluation over $GF(2^{d})$, having cell size $w$, worst case update time $t_{u}$ and expected average query time $t_{q}$ must satisfy:

[TABLE]

Note that this lower bound is not restricted to have $d=O(\lg n)$ (corresponding to having polynomially many queries). It holds for arbitrarily large $d$ and thus demonstrates that our lower bound actually grows as log of the number of queries, times a $\sqrt{\lg n}$. At least up to a certain (unavoidable) barrier (the $\sqrt{n}$ bound in the min is precisely when the query time is large enough that the data structure can read all cells associated to more than half of the epochs). We remark that the majority of previous lower bound techniques could also replace a $\lg n$ in the lower bounds by a $d$ for problems with $2^{d}$ queries. Our introduction focuses on the most natural case of polynomially many queries ($d=\Theta(\lg n)$) for ease of exposition.

Polynomial evaluation has been studied quite intensively from a lower bound perspective, partly since it often allows for very clean proofs. The previous work on the problem considered the standard (non-boolean) version in which we are required to output the value $P(x)$, not just its least significant bit. Miltersen [Mil95] first considered the static version where the polynomial is given in advance and we disallow updates. He proved a lower bound of $t_{q}=\Omega(d/\lg S)$ where $S$ is the space usage of the data structure in number of cells. This was improved by Larsen [Lar12b] to $t_{q}=\Omega(d/\lg(Sw/(nd)))$, which remains the highest static lower bound proved to date. Note that the lower bound peaks at $t_{q}=\Omega(d)$ for linear space $S=O(nd/w)$. Larsen [Lar12b] also extended his lower bound to the dynamic case (though for a slightly different type of updates), resulting in a lower bound of $t_{q}=\Omega(d\lg n/(\lg(wt_{u})\cdot\lg(wt_{u}/d))$. Note that none of these lower bounds are greater than $t_{q}=\Omega(\lg n/\lg t_{u})$ per output bit and in that sense they are much weaker than our new lower bound.

In [GM07], Gál and Miltersen considered succinct data structures for polynomial evaluation. Succinct data structures are data structures that use space close to the information theoretic minimum required for storing the input. In this setting, they showed that any data structure for polynomial evaluation must satisfy $t_{q}r=\Omega(n)$ when $2^{d}\geq(1+\epsilon)n$ for any constant $\epsilon>0$. Here $r$ is the redundancy, i.e. the additive number of extra bits of space used by the data structure compared to the information theoretic minimum. Note that even for data structures using just a factor $2$ more space than the minimum possible, the time lower bound reduces to the trivial $t_{q}=\Omega(1)$. For data structures with non-determinism (i.e., they can guess the right cells to probe), Yin [Yin10] proved a lower bound matching that of Miltersen.

On the upper bound side, Kedlaya and Umans [KU08] presented a word-RAM data structure for the static version of the problem, having space usage $n^{1+\epsilon}d^{1+o(1)}$ and worst case query time $\lg^{O(1)}n\cdot d^{1+o(1)}$, getting rather close to the lower bounds. While not discussed in their paper, a simple application of the logarithmic method makes their data structure dynamic with an amortized update time of $n^{\epsilon}d^{1+o(1)}$ and worst case query time $\lg^{O(1)}n\cdot d^{1+o(1)}$.

Parity Searching in Butterfly Graphs.

In a seminal paper [Pǎt08], Pǎtraşcu presented an exciting connection between an entire class of data structure problems. Starting from a problem of reachability oracles in the Butterfly graph, he gave a series of reductions to classic data structure problems. His reductions resulted in $t_{q}=\Omega(\lg n/\lg(Sw/n))$ lower bounds for static data structures solving any of these problems.

We modify Pǎtraşcu’s reachability problem such that we can use it in reductions to prove new dynamic lower bounds. In our version of the problem, which we term parity searching in Butterfly graphs, the data structure must maintain a set of directed acyclic graphs (Butterfly graphs of the same degree $B$, but different depths) under updates which assign binary weights to edges, and support queries that ask to compute the parity of weights assigned to edges along a number of paths in these graphs. The formal definition of this version of the problem is deferred to Section 6.2.

While this new problem might sound quite artificial and incompatible to work with, we show that parity searching in Butterfly graphs in fact reduces to many classic problems, hence proving lower bounds on this problem is the key to many of our results. Indeed, our starting point is the following lower bound:

Theorem 3**.**

Any dynamic data structure for parity searching in Butterfly graphs of degree $B=(wt_{u})^{8}$, with a total of $n$ edges, having cell size $w$, worst case update time $t_{u}$ and expected average query time $t_{q}$ must satisfy:

[TABLE]

In the remainder of this section, we present new lower bounds which we derive via reductions from parity searching in Butterfly graphs . For context, our results are complemented with a discussion of previous work.

2D Range Counting.

In 2D range counting, we are given $n$ points $P$ on a $[U]\times[U]$ integer grid, for some $U=n^{O(1)}$. We must preprocess the points such that given a query point $q=(x,y)\in[U]\times[U]$, we can return the number of points $p\in P$ that are dominated by $q$ (i.e. $p.x\leq q.x$ and $p.y\leq q.y$). In the dynamic version of the problem, an update specifies a new point to insert. 2D range counting is a fundamental problem in both computational geometry and spatial databases and many variations of it have been studied over the past many decades.

Via a reduction from reachability oracles in the Butterfly graph, Pǎtraşcu [Pǎt08] proved a static lower bound of $t_{q}=\Omega(\lg n/\lg(Sw/n))$ for this problem, even in the case where one needs only to return the parity of the number of points dominated by $q$. Recall that this is the 2D range parity problem.

It turns out that a fairly easy adaptation of Pǎtraşcu’s reduction implies the following:

Theorem 4**.**

Any dynamic cell probe data structure for 2D range parity, having cell size $w$, worst case update time $t_{u}$ and expected query time $t_{q}$, gives a dynamic cell probe data structure for parity searching in Butterfly graphs (for any degree $B$) with cell size $w$, worst case update time $O(t_{u})$ and average expected query time $t_{q}$.

Combining this with our lower bound for parity searching in Butterfly graphs (Theorem 3), we obtain:

Corollary 1**.**

Any cell probe data structure for 2D range parity, having cell size $w$, worst case update time $t_{u}$ and expected query time $t_{q}$ must satisfy:

[TABLE]

In addition to Pǎtraşcu’s static lower bound, Larsen [Lar12a] studied the aforementioned variant of the range counting problem, called 2D range sum, in which points are assigned $\Theta(\lg n)$-bit integer weights and the goal is to compute the sum of weights assigned to points dominated by $q$. As previously discussed, Larsen’s lower bound for dynamic 2D range sum was $t_{q}=\Omega((\lg n/\lg(t_{u}w))^{2})$ and was the first lower bound to break the $\Omega(\lg n)$-barrier, though only for a problem with $\Theta(\lg n)$ bit output. Weinstein and Yu [WY16] later re-proved Larsen’s lower bound, this time extending it to the setting of amortized update time and a very high probability of error. Note that these lower bounds remain below the logarithmic barrier when measured per output bit of a query. While 2D range counting (not the parity version) also has $\Theta(\lg n)$-bit outputs, it seems that the techniques of Larsen and Weinstein and Yu are incapable of proving an $\omega(\lg n)$ lower bound for it. Thus the strongest previous lower bound for the dynamic version of 2D range counting is just the static bound of $t_{q}=\Omega(\lg n/\lg(t_{u}w))$ (since one cannot build a data structure with space usage higher than $S=t_{u}n$ in $n$ operations). As a rather technical explanation for why the previous techniques fail, it can be observed that they all argue that a collection of $m=n/\mathrm{poly}(\lg n)$ queries have $\Omega(m\lg n)$ bits of entropy in their output. But for 2D range counting, having $n/\mathrm{poly}(\lg n)$ queries means that on average, each query contains just $\mathrm{poly}(\lg n)$ new points, reducing the total entropy to something closer to $O(m\lg\lg n)$. This turns out to be useless for the lower bound arguments. It is conceivable that a clever argument could show that the entropy remains $\Omega(m\lg n)$, but this has so forth resisted all attempts.

From the upper bound side, JáJá, Mortensen and Shi [JMS04] gave a static 2D range counting data structure using linear space and $O(\lg n/\lg\lg n)$ query time, which is optimal by Pǎtraşcu’s lower bound. For the dynamic case, Brodal et al. [BGJS11] gave a data structure with $t_{q}=t_{u}=O((\lg n/\lg\lg n)^{2})$. Our new lower bound shrinks the gap between the upper and lower bound on $t_{q}$ to only a factor $\sqrt{\lg n}\lg\lg n$ for $t_{u}=\mathrm{poly}(\lg n)$.

2D Rectangle Stabbing.

In 2D rectangle stabbing, we must maintain a set of $n$ 2D axis aligned rectangles with integer coordinates, i.e. rectangles are of the form $[x_{1},x_{2}]\times[y_{1},y_{2}]$. We assume coordinates are bounded by a polynomial in $n$. An update inserts a new rectangle. A query is specified by a point $q$, and one must return the number of rectangles containing $q$. This problem is known to be equivalent to 2D range counting via a folklore reduction. Thus all the bounds in the previous section, both upper and lower bounds, also apply to this problem. Furthermore, 2D range parity is also equivalent to 2D rectangle parity, i.e. returning just the parity of the number of rectangles stabbed.

Range Selection and Range Median.

In range selection, we are to store an array $A=\{A[0],\dots,A[n-1]\}$ where each entry stores an integer bounded by a polynomial in $n$. A query is specified by a triple $(i,j,k)$. The goal is to return the index of the $k$’th smallest entry in the subarray $\{A[i],\dots,A[j]\}$. In the dynamic version of the problem, entries are initialized to [math]. Updates are specified by an index $i$ and a value $a$ and has the effect of changing the value stored in entry $A[i]$ to $a$. In case of multiple entries storing the same value, we allow returning an arbitrary index being tied for $k$’th smallest.

We give a reduction from parity searching in Butterfly graphs:

Theorem 5**.**

Any dynamic cell probe data structure for range selection, having cell size $w$, worst case update time $t_{u}$ and expected query time $t_{q}$, gives a dynamic cell probe data structure for parity searching in Butterfly graphs (for any degree $B$) having cell size $w$, worst case update time $O(t_{u}\lg^{2}n)$ and expected average query time $t_{q}$. Furthermore, this holds even if we force $i=0$ in queries $(i,j,k)$ and require only that we return whether the $k$’th smallest element in $A[0],\dots,A[j]$ is stored at an even or odd position.

Since we assume $w=\Omega(\lg n)$, combining this with Theorem 3 immediately proves the following:

Corollary 2**.**

Any cell probe data structure for range selection, having cell size $w$, worst case update time $t_{u}$ and expected query time $t_{q}$ must satisfy:

[TABLE]

Furthermore, this holds even if we force $i=0$ in queries $(i,j,k)$ and require only that we return whether the $k$’th smallest element in $A[0],\dots,A[j]$ is stored at an even or odd position.

While range selection is not a boolean data structure problem, it is still a fundamental problem and for the same reasons as mentioned under 2D range counting, the previous lower bound techniques seem incapable of proving $\omega(\lg n)$ lower bounds for the dynamic version. Thus we find our new lower bound very valuable despite the problem not beeing boolean . Also, we do in fact manage to prove the same lower bound for the boolean version where we need only determine whether the index of the $k$’th smallest element is even or odd.

For the static version of the problem, Jørgensen and Larsen [JL11] proved a lower bound of $t_{q}=\Omega(\lg n/\lg(Sw/n))$. Their proof was rather technical and a new contribution of our work is that their static lower bound now follows by reduction also from Pǎtraşcu’s lower bound for reachability oracles in the Butterfly graph. For the dynamic version of the problem, no lower bound stronger than the $t_{q}=\Omega(\lg n/\lg(t_{u}w))$ bound following from the static bound was previously known.

On the upper bound side, Brodal et al. [BGJS11] gave a linear space static data structure with query time $t_{q}=O(\lg n/\lg\lg n)$. This matches the lower bound of Jørgensen and Larsen. They also gave a dynamic data structure with $t_{q}=t_{u}=O((\lg n/\lg\lg n)^{2})$.

Since we prove our lower bound for the version of range selection where $i=0$, also known as prefix selection, we can re-execute a reduction of Jørgensen and Larsen [JL11]. This means that we also get a lower bound for the fundamental range median problem. Range median is the natural special case of range selection where $k=\lceil(j-i+1)/2\rceil$.

Corollary 3**.**

Any cell probe data structure for range median, having cell size $w$, worst case update time $t_{u}$ and expected query time $t_{q}$ must satisfy:

[TABLE]

Furthermore, this holds even if we are required only to return whether the median amongst $A[i],\dots,A[j]$ is stored at an even or odd position.

We note that the upper bound of Brodal et al. for range selection is also the best known upper bound for range median.

2 Organization

In Section 3 we introduce both the dynamic cell probe model and the one-way communication model, which is the main proxy for our results. In Section 4 we state the formal version of Theorem 1 and give its proof as well as the proof of the Peak-to-Average lemma. Section 5 and onwards are devoted to applications of our new simulation theorem, starting with a lower bound for polynomial evaluation. In Section 6 we formally define parity searching in Butterfly graphs and prove a lower bound for it using our simulation theorem. Finally, Section 7 presents a number of reductions from parity searching in Butterfly graphs to various fundamental data structure problems, proving the remaining lower bounds stated in the introduction.

3 Preliminaries

The dynamic cell probe model.

A dynamic data structure in the cell probe model consists of an array of memory cells, each of which can store $w$ bits. Each memory cell is identified by a $w$-bit address, so the set of possible addresses is $[2^{w}]$. It is natural to assume that each cell has enough space to address (index) all update operations performed on it, hence we assume that $w=\Omega(\lg n)$ when analyzing a sequence of $n$ operations.

Upon an update operation, the data structure can perform read and write operations to its memory so as to reflect the update, by probing a subset of memory cells. This subset may be an arbitrary function of the update and the content of the memory cells previously probed during this process. The update time of a data structure, denoted by $t_{u}$, is the number of probes made when processing an update (this complexity measure can be measured in worst-case or in an amortized sense). Similarly, upon a query operation, the data structure performs a sequence of probes to read a subset of the memory cells in order to answer the query. Once again, this subset may by an arbitrary (adaptive) function of the query and previous cells probed during the processing of the query. The query time of a data structure, denoted by $t_{q}$, is the number of probes made when processing a query.

3.1 One-way protocols and “Epoch” communication games

A useful way to abstract the information-theoretic bottleneck of dynamic data structures is communication complexity. Our main results (both upper and lower bounds) are cast in terms of the following two-party communication games, which are induced by dynamic data structure problems:

Definition 1** (Epoch Communication Games $G^{i}_{\mathcal{P}}$).**

Let $\mathcal{P}$ be a dynamic data structure problem, consisting of a sequence of $n$ update operations divided into epochs $\mathcal{U}=(\mathbf{U}_{\ell},\mathbf{U}_{l-1},\ldots,\mathbf{U}_{1})$, where $|\mathbf{U}_{i}|=n_{i}$ (and $\sum_{i}n_{i}=n$), followed by a single query $q\in\mathcal{Q}$. For each epoch $i\in[\ell]$, the two-party communication game $G^{i}_{\mathcal{P}}$ induced by $\mathcal{P}$ is defined as follows:

• •

Alice receives all update operations $\mathcal{U}=(\mathbf{U}_{\ell},\mathbf{U}_{l-1},\ldots,\mathbf{U}_{1})$.

• •

Bob receives $\mathbf{U}_{-i}:=\mathcal{U}\setminus\{\mathbf{U}_{i}\}$ (i.e., all updates except those in epoch $i$) and a query $q\in\mathcal{Q}$ for $\mathcal{P}$.

• •

The goal of the players is to output the correct answer to $q$, that is, to output $\mathcal{P}(q,\mathcal{U})$.

We shall consider the following restricted model of communication for solving such communication games.

Definition 2** (One-Way Randomized Communication Protocols).**

Let $f:\mathcal{X}\times\mathcal{Y}\mapsto\{0,1\}$ be a two-party boolean function. A one-way communication protocol $\pi$ for $f(x,y)$ under input distribution $\mu$ proceeds as follows:

• •

Alice and Bob have shared access to a public random string $R$ of their choice.

• •

Alice sends Bob a single message, $M_{A}(x,R)$, which is only a function of her input and the public random string.

• •

*Based on Alice’s message, Bob must output a value $v_{\pi}=v_{\pi}(y,R,M_{A})\in\{0,1\}$. *

We say that $\pi$ $\epsilon$-solves $f$ under $\mu$ with cost $m$, if :

• •

For any input $x$, Alice never sends more than $m$ bits to Bob, i.e., $|M_{A}(x,R)|\leq m$, for all $x,r$.

• •

$\Pr_{(x,y)\sim\mu,R}[v_{\pi}=f(x,y)]\geq 1/2+\epsilon.$**

Let us denote by

[TABLE]

the largest advantage $\epsilon$ achievable for predicting $f$ under $\mu$ via an $m$-bit one-way communication protocol. For example, when applied to the boolean communication problem $G^{i}_{\mathcal{P}}$, we say that $G^{i}_{\mathcal{P}}$ has an $m$-bit one-way communication protocol with advantage $\epsilon$, if $\overrightarrow{\mathsf{adv}}(G^{i}_{\mathcal{P}},\mu,m)\geq\epsilon$. We remark that we sometimes use the notation $\|\pi\|$ to denote the message-length (i.e., number of bits $m$) of the communication protocol $\pi$.

4 One-Way Weak Simulation of Dynamic Data Structures

In this section we prove our main result, Theorem 1. For any dynamic decision problem $\mathcal{P}$, we show that if $\mathcal{P}$ admits an efficient data structure $D$ with respect to a random sequence of $n$ updates divided into $\ell:=\lg_{\beta}n$ epochs $\mathcal{U}=(\mathbf{U}_{\ell},\mathbf{U}_{\ell-1},\ldots,\mathbf{U}_{1})$, then we can use it to devise an efficient one-way communication protocol for the underlying two-party communication problem $G^{i}_{\mathcal{P}}$ of some (large enough) epoch $i$, with a nontrivial success (advantage over random guessing).

Throughout this section, let us denote the size of epoch $i$ by $n_{i}:=|\mathbf{U}_{i}|=\beta^{i}$, where we require $\beta=(w\cdot t_{u})^{\Theta(1)}$, and $\sum_{i=1}^{\ell}n_{i}=n$. We prove the following theorem.

Theorem 1 (restated).

Let $\mathcal{P}$ be a dynamic boolean data structure problem, with $n$ random updates grouped into epochs $\mathcal{U}=\{\mathbf{U}_{i}\}_{i=1}^{\ell}$, such that $|\mathbf{U}_{i}|=\beta^{i}$, followed by a single query $q\in\mathcal{Q}$. If $\mathcal{P}$ admits a dynamic data structure $D$ with worst-case update time $t_{u}$ and average (over $\mathcal{Q}$) expected query time $t_{q}$ satisfying $t_{q}(w\cdot t_{u})^{a+1}\leq n^{1/2}$, then there exists some epoch $i\in[\ell/2,\ell]$ for which

[TABLE]

as long as $\beta=(w\cdot t_{u})^{\Theta(1)}\geq(w\cdot t_{u})^{a}$ for a constant $a>1$.

Proof.

Consider the memory state $M=M(\mathcal{U})$ of $D$ after the entire update sequence $\mathcal{U}$, and for each cell $c\in M$, define its associated epoch $E(c)$ to be the last epoch in $[\ell]$ during which $c$ was probed (note that $E(c)$ is a random variable over the random update sequence $\mathcal{U}$). For each query $q\in\mathcal{Q}$, let $T_{q}$ be the random variable denoting the number of probes made by $D$ on query $q$ (on the random update sequence). For each query $q$ and epoch $i$, let $T_{q}^{i}$ denote the number of probes on query $q$ to cells associated with epoch $i$ (i.e., cells $c$ for which $E(c)=i$).

By definition, we have $\frac{1}{|\mathcal{Q}|}\sum_{q\in\mathcal{Q}}\mathop{{}\mathbb{E}}[T_{q}]=t_{q}$ and $T_{q}=\sum_{i=1}^{\ell}T_{q}^{i}$. By averaging, there is an epoch $i\in[\ell/2,\ell]$ such that $\frac{1}{|\mathcal{Q}|}\sum_{q\in\mathcal{Q}}\mathop{{}\mathbb{E}}[T_{q}^{i}]\leq 2t_{q}/\ell$. By Markov’s inequality and a union bound, there exists a subset $\mathcal{Q}^{\prime}\subseteq\mathcal{Q}$ of $|\mathcal{Q}|/2$ queries such that both

[TABLE]

for every query $q\in\mathcal{Q}^{\prime}$. By Markov’s inequality and union bound, for each $q\in\mathcal{Q}^{\prime}$, we have

[TABLE]

Note that, while Bob cannot identify the event “$T_{q}^{i}\leq 32t_{q}/\ell,T_{q}\leq 32t_{q}$” (as it depends on Alice’s input as well), he does know whether his query $q$ is in $\mathcal{Q}^{\prime}$ or not, which is enough to certify (2).

Now, suppose that Alice samples each cell associated with epoch $i$ in $M$ independently with probability $p$, where

[TABLE]

(note that, by definition of $G^{i}_{\mathcal{P}}$, Alice can indeed generate the memory state $M$ and compute the associated epoch for each cell, as her input consists of the entire update sequence). Let $C_{0}$ be the resulting set of cells sampled by Alice. Alice sends Bob $C_{0}$ (both addresses and contents). For a query $q\in\mathcal{Q}^{\prime}$, let $\mathcal{W}_{q}$ denote the event that the set of cells $C_{0}$ Bob receives, contains all $T_{q}^{i}$ cells associated with epoch $i$ and probed by the data structure. By Equation (2), we have that for every $q\in\mathcal{Q}^{\prime}$

[TABLE]

If Bob could detect the event $\mathcal{W}_{q}$, we would be done. Indeed, let $C_{2}$ denote the set of (addresses and contents of) cells associated with all future epochs $j<i$, i.e., all the cells probed by $D$ succeeding epoch $i$. Due to the geometrically decreasing sizes of epochs, sending $C_{2}$ requires less than $n_{i}/(w\cdot t_{u})^{a-1}$ bits of communication. Since Bob has all the updates preceding epoch $i$, he can simulate the data structure and generate the correct memory state of $D$ right before epoch $i$. In particular, Bob knows for every cell, assuming it is not probed since epoch $i$ (thus associated with some epoch $j>i$), what its content will be. Therefore, when he is further given the messages $(c_{0},c_{2})$, Bob would be able to simulate the data structure perfectly on query $q$, assuming the event $\mathcal{W}_{q}$ occurs. If Bob could detect $\mathcal{W}_{q}$, he could simply output a random bit if it does not occur, and follow the data structure if it does. This strategy would have already produced an advantage of $p^{32t_{q}/\ell}\geq\exp(-t_{q}\lg^{2}(w\cdot t_{u})/\lg n)$, which would have finished the proof. As explained in the introduction, Bob has no hope of certifying the occurrence of the event $\mathcal{W}_{q}$, hence we must take a fundamentally different approach for arguing that condition (3) can nevertheless be (implicitly) used to devise a strategy for Bob with a nontrivial advantage. This is the heart of the proof.

To this end, note that, given a query $q\in\mathcal{Q}^{\prime}$, a received sample $c_{0}$ and all cells $c_{2}$ associated with some epoch $j<i$, Bob can simulate $D$ on his partial update sequence ($u_{-i}$), filling in the memory updates according to $c_{0}$ and $c_{2}$, and pretending that all cells in the query-path of $q$ which are associated with epoch $i$ are actually sampled in $c_{0}$ (i.e., pretending that the event $\mathcal{W}_{q}$ occurs). See Step 5 of Figure 1 for the formal simulation argument. Let $M^{\prime}(u_{-i},c_{0},c_{2})$ denote the resulting memory state obtained by Bob’s simulation in the figure, given $u_{-i}$ and his received sets of cells $c_{0},c_{2}$.

Now, let us consider the (deterministic) sequence of cells $S_{q}$ that $D$ would probe given query $q$ in the above simulation with respect to Bob’s memory state $M^{\prime}(u_{-i},c_{0},c_{2})$. Let us say that the triple $(u_{-i},c_{0},c_{2})$ is good for a query $q\in\mathcal{Q}^{\prime}$, if $\Pr_{\mathbf{U}_{i}}[\mathcal{W}_{q}|\mathbf{U}_{-i}=u_{-i},C_{0}=c_{0},C_{2}=c_{2}]\geq p^{32t_{q}/\ell}/4$ and $|S_{q}|\leq 32t_{q}$. That is, $(u_{-i},c_{0},c_{2})$ is good for $q$, if the posterior probability of $\mathcal{W}_{q}$ is (relatively) high and $S_{q}$ is not too large. By Equation (3) and Markov’s inequality, the probability that the triple $(u_{-i},c_{0},c_{2})$ satisfies $\Pr_{\mathcal{U}}[\mathcal{W}_{q},T_{q}\leq 32t_{q}|u_{-i},c_{0},c_{2}]\geq p^{32t_{q}/\ell}/4$, is at least $p^{32t_{q}/\ell}/4$ (indeed, the expectation in (3) can be rewritten as $\mathop{{}\mathbb{E}}_{\mathbf{U}_{-i},C_{0},C_{2}}\Pr_{\mathbf{U}_{i}}[\mathcal{W}_{q},T_{q}\leq 32t_{q}|\mathbf{U}_{-i},C_{0},C_{2}]$, since $C_{2}$ is a deterministic function of $\mathcal{U}$). Note that when $\mathcal{W}_{q}$ occurs, the value of $T_{q}$ is completely determined given $u_{-i}$, $c_{0}$ and $c_{2}$, in which case $|S_{q}|=T_{q}$, and thus the probability that $(u_{-i},c_{0},c_{2})$ is good is at least $p^{32t_{q}/\ell}/4$. From now on, let us focus only on the case that $(u_{-i},c_{0},c_{2})$ that Alice sends is good, since Bob can identify whether $u_{-i},c_{0},c_{2}$ is good based on $q$ and Alice’s message, and if it is not, he will output a random bit.

We caution that $S_{q}$ is simply a set of memory addresses in $M$, not necessarily the correct one – in particular, while the addresses of the cells $S_{q}$ are determined by the above simulation, the contents of these cells (in $M$) are not – they are a random variable of $\mathbf{U}_{i}$, as the sample $c_{0}$ is very unlikely to contain all the associated cells). For any assignment $z\in[2^{w}]^{S_{q}}$ to the contents of the cells in $S_{q}$, let us denote by

[TABLE]

the probability that the memory content of the sequence of cells $S_{q}$ is equal to $z$, conditioned on $u_{-i},c_{0},c_{2}$.

Every content assignment $Z=z$ to $S_{q}$, generates some posterior distribution on the correct query path (i.e., with respect to the true memory state $M$) and therefore on the output $\mathcal{P}(q,\mathcal{U})$ of the query $q$ with respect to $\mathcal{U}$. Hence we may look at the joint probability distribution of the event “$\mathcal{P}(q,\mathcal{U})=1$” and the assignment $Z$ which is

[TABLE]

Now, consider the function

[TABLE]

Equivalently, conditioned on $u_{-i}$, $c_{0}$ and $c_{2}$, $f(z)$ is the bias of the random varaible $\mathcal{P}(q,\mathcal{U})$ conditioned on $S_{q}\leftarrow z$, multiplied by the probability of $S_{q}\leftarrow z$.

Note that, since $\eta_{q}(z)\leq\mu_{q}(z)$ for every assignment $z$, we have $|f(z)|\leq\mu_{q}(z)/2$, and since $\mu_{q}(z)$ is a probability distribution, this fact implies that: (i) $\sum_{z}|f(z)|\leq\frac{1}{2}$. Furthermore, we shall argue that $\Pr[\mathcal{W}_{q}\mid u_{-i},c_{0},c_{2}]\geq p^{32t_{q}/\ell}/4$ (as we always condition on good $u_{-i},c_{0},c_{2}$), in which case the contents of $S_{q}$ are completely determined by $u_{-i},c_{0},c_{2}$ (we postpone the formal argument to the Analysis section below). Denoting by $z^{*}$ the content assignment to $S_{q}$ induced by $u_{-i},c_{0},c_{2}$, we observe that conditioned on $\mathcal{W}_{q}$, $S_{q}$ will be precisely the correct set of cells probed by $D$ on $q$, in which case $\mathcal{P}(q,\mathcal{U})$ is determined by $z^{*},q,u_{-i},c_{0},c_{2}$. Formally, this fact means that: (ii) $|f(z^{*})|=\frac{1}{2}\cdot\Pr[S_{q}\leftarrow z^{*}\mid u_{-i},c_{0},c_{2}]\geq\Omega(p^{32t_{q}/\ell})$.

Conditions (i)$+$(ii) above imply that $f=f^{q}_{u_{-i},c_{0},c_{2}}$ satisfies the premise of the Peak-to-Average Lemma (Lemma 1) with $\Sigma:=[2^{w}],k:=O(t_{q}),\epsilon:=\Omega(p^{32t_{q}/\ell})=\exp(-t_{q}\lg^{2}(w\cdot t_{u})/\lg n)$. Recall that the lemma guarantees there is a not-too-large subset $Y\subseteq S_{q}$ of coordinates ($=$ addresses) of $S_{q}$, which Bob can privately compute,555Indeed, $Y$ is only a function of $q$, $f^{q}_{u_{-i},c_{0},c_{2}}$, $c_{0}$,$c_{2}$ and the prior distribution on $\mathcal{U}$, and Bob possesses all this information. such that if the values of the coordinates in $Y$ are also revealed, then the conditional expectation of $f^{q}_{u_{-i},c_{0},c_{2}}$ $\left(\text{namely, }\mathop{{}\mathbb{E}}_{S_{q}|_{Y}}\left|\Pr_{\mathbf{U}_{i}}\left[\mathcal{P}(q,\mathcal{U})=1\mid u_{-i},c_{0},c_{2},S_{q}|_{Y}\right]\;-\;1/2\right|\right)$, which is the average of Bob’s “maximum-likelihood” estimate for $\mathcal{P}(q,\mathcal{U})$, is non-negligible (the formal details are postponed to the Analysis section below).

Given this insight, a natural strategy for the players is for Alice to further send Bob the contents of cells in the subset $Y$. While Alice does not know the subset $Y$,666Indeed, $Y$ is a function of $q$. she can use public randomness to sample yet another random set $C_{1}$ of cells from the entire memory $M$, where now every cell is sampled with equal probability $p$, and send the subset of $C_{1}$ that is associated with epoch $i$ to Bob. (Note that it is important that this time the players use public randomness to subsample from the entire memory state $M$, since Alice does not know $Y$ and yet Bob must be absolutely certain that all cells in $Y$ were subsampled. Notwithstanding, to keep communication low, it is crucial that Alice sends Bob only the contents of cells associated with epoch $i$). Since $|Y|$ is guaranteed to be relatively small (of order $O(\sqrt{k\lg(1/\epsilon)})$), the probability $p^{|Y|}$ that all cells in $Y$ get sampled will be sufficiently noticeable, in which case we shall argue that Bob’s maximum-likelihood strategy will output the correct answer $\in\{0,1\}$ with the desired nontrivial advantage. The formal one-way protocol $\pi$ that the parties execute is described in Figure 1.

Analysis.

We now turn to the formal analysis of the protocol $\pi$. We need to show

• •

(Communication cost) $\|\pi\|\leq O(n_{i}/(w\cdot t_{u})^{a-1})$ .

• •

(Correctness) $\Pr_{G^{i}_{\mathcal{P}}\sim\mathcal{U},q\in_{R}\mathcal{Q}}\left[\pi(G^{i}_{\mathcal{P}})=\mathcal{P}(q,\mathcal{U})\right]\geq 1/2+\exp\left(-t_{q}\lg^{2}(w\cdot t_{u})/\sqrt{\lg n}\right)$.

Communication.

In both Step 2 and Step 3, Alice sends at most $2p|\mathbf{U}_{i}|t_{u}\cdot(2w)+1$ bits. In Step 4, Alice sends at most $|\mathbf{U}_{<i}|\cdot t_{u}\cdot(2w)=\sum_{j<i}|\mathbf{U}_{j}|\cdot t_{u}\cdot(2w)$ bits. Since $|\mathbf{U}_{j}|=n_{j}=\beta^{j}$, the total communication cost is at most

[TABLE]

Correctness.

Let $\pi^{\prime}$ be the variant of the protocol $\pi$ in which, when executing Step 2 and Step 3, Alice ignores the condition of whether the samples $C_{0}$ or $C_{1}$ exceed the specified size limit, i.e., she always sends a bit 1 followed by all sampled cells. For simplicity of analysis, we will first show that $\pi^{\prime}$ has the claimed success probability, and then show that the impact of the above event (i.e., conditioning on $C_{0}$ and $C_{1}$ being within the size bound) is negligible, as it occurs with extremely high probability.

We first claim that the probability (over $\mathcal{U}$ and an average query $q\in_{R}Q$) that $\pi^{\prime}$ reaches Step $6$ is not too small. By (1) and Markov’s inequality, and by the discussion below (3), the probability that $q\in\mathcal{Q}^{\prime}$ and $(u_{-i},c_{0},c_{2})$ is “good” for $q$ is at least $\Omega(p^{32t_{q}/\ell})\geq\exp(-t_{q}\lg^{2}(w\cdot t_{u})/\lg n)$. This is precisely the probability that $\pi^{\prime}$ reaches Step 6.

We now calculate the success probability of $\pi^{\prime}$ conditioned on reaching Step 6. To this end, fix a set $Y\subseteq S_{q}$ of size $\kappa$. Then by Step $7$, the success probability of $\pi^{\prime}$ conditioned on $u_{-i},c_{0},c_{2}$ and the event “$Y\subseteq C_{1}$” is

[TABLE]

where the last transition is by the definition of $f^{q}_{u_{-i},c_{0},c_{2}}$ in (4). Note that for any $z$, it holds that $|f(z)|\leq\frac{1}{2}\cdot\Pr[S_{q}\leftarrow z\mid u_{-i},c_{0},c_{2}]$. Thus, $\sum_{z\in[2^{w}]^{S_{q}}}|f(z)|\leq\frac{1}{2}$. On the other hand, since we always condition on good $(u_{-i},c_{0},c_{2})$, we have $\Pr[\mathcal{W}_{q}\mid u_{-i},c_{0},c_{2}]\geq p^{32t_{q}/\ell}/4$. That is, with probability at least $p^{32t_{q}/\ell}/4$ all cells in $S_{q}$ associated with epoch $i$ are contained in $c_{0}$. In this case, the contents of $S_{q}$ are completely determined by $u_{-i},c_{0},c_{2}$. Indeed, the contents of the cells associated with epoch $<i$ are determined by $c_{2}$; the cells associated with epoch $i$ are determined by $c_{0}$; the remaining cells are determined by $u_{>i}$. Let $z^{*}$ denote the assignment to $S_{q}$, induced by $u_{-i}$ and the contents of $c_{0},c_{2}$ conditioned on the occurrence of $\mathcal{W}_{q}$. By the definition of $S_{q}$, when $\mathcal{W}_{q}$ happens, $S_{q}$ will be exactly the set of cells the data structure probes. Thus, the output of $q$ is also determined. We therefore have $|f(z^{*})|=\frac{1}{2}\cdot\Pr[S_{q}\leftarrow z^{*}\mid u_{-i},c_{0},c_{2}]\geq\Omega(p^{32t_{q}/\ell})$. We conclude that the function $f=f^{q}_{u_{-i},c_{0},c_{2}}$ satisfies the premise of the Peak-to-Average lemma (Lemma 1) with

• •

$\Sigma=[2^{w}]$;

• •

$k=|S_{q}|\leq O(t_{q})$;

• •

$\epsilon=p^{32t_{q}/\ell}/4\geq\exp(-t_{q}\lg^{2}(w\cdot t_{u})/\lg n)$.777We used the fact that $\ell=\Theta(\lg_{\beta}n)$ and $\beta=(w\cdot t_{u})^{\Theta(1)}$.

Without loss of generality, we may assume $\lg(w\cdot t_{u})\ll\sqrt{\lg n}$, and thus $\epsilon\in[2^{-O(k)},1]$.888In fact, if $\lg(w\cdot t_{u})\geq\Omega(\sqrt{\lg n})$, the right-hand side of the inequality in the theorem statement is less than $p^{t_{q}}$, hence the statement becomes trivial. Indeed, with probability $p^{t_{q}}$, Alice samples all cells probed by the data structure on query $q$. Therefore, the lemma guarantees there is a set $Y\subset S_{q}$ of cells that has size at most

[TABLE]

for which

[TABLE]

This justifies Step 6 of the protocol. It follows that, for any $q\in\mathcal{Q}^{\prime}$, the probability that the sample $C_{1}$ of cells contains the set $Y$ is at least

[TABLE]

Equation (4) therefore implies that, conditioned on the event that $|Y|\subseteq C_{1}$, the probability that $\pi^{\prime}$ outputs a correct answer is

[TABLE]

and combining this with (6) and the probability that $\pi^{\prime}$ reaches Step 6, we conclude that the overall success probability of $\pi$, conditioned on the protocol not aborting when $c_{0}$ or $c_{1}$ is too large, is

[TABLE]

To finish the proof, it therefore suffices to argue that the probability that $\pi$ aborts due to this event is tiny. To this end, let $A_{i}$ denote the random variable representing the number of associated cells with epoch $i$. We know that $A_{i}\leq|\mathbf{U}_{i}|\cdot t_{u}=n_{i}\cdot t_{u}$ (since the worst-case update time of $D$ is $t_{u}$ by assumption). Now, let $\mathcal{E}_{0}$ denote the event that Alice’s sample in Step 2 of the protocol is too large, i.e., that $``|C_{0}|>2p|\mathbf{U}_{i}|\cdot t_{u}"$. Similarly, let $\mathcal{E}_{1}$ denote the event that in Step 3 of the protocol, $``|C_{1}|>2p|\mathbf{U}_{i}|\cdot t_{u}"$. Denote $\mathcal{E}:=\mathcal{E}_{0}\vee\mathcal{E}_{1}$ (note that this is the event (ii) in Step 5 of $\pi$). Since both sets $C_{0}$ and $C_{1}$ are i.i.d samples where each cell is sampled independently with probability $p$, a standard Chernoff bound implies that

[TABLE]

Finally, since $i\geq\ell/2$ and thus $n_{i}\geq n^{1/2}\geq t_{q}(w\cdot t_{u})^{a+1}$, by (7), (8) and a union bound, we conclude that

[TABLE]

which completes the proof of the entire theorem.

∎

While Theorem 1 is very clean, we shall need a slightly more technical version of it for some of our lower bound proofs. The following corollary follows directly by examining the proof of Theorem 1:

Theorem 6** (One-Way Weak Simulation of Epoch $i$).**

Let $\mathcal{P}$ be a dynamic boolean data structure problem, with $n$ random updates grouped into epochs $\mathcal{U}=\{\mathbf{U}_{i}\}_{i=1}^{\ell}$, such that $|\mathbf{U}_{i}|=\beta^{i}$ followed by a single query $q\in\mathcal{Q}$. If $\mathcal{P}$ admits a dynamic data structure $D$ with worst-case update time $t_{u}$ and average (over $\mathcal{Q}$) expected query time $t_{q}$, such that for some epoch $i\in[\ell/2,\ell]$ it holds that

[TABLE]

then if $t_{q}(w\cdot t_{u})^{a+1}\leq n_{i}$, we have

[TABLE]

as long as $\beta=(w\cdot t_{u})^{\Theta(1)}\geq(w\cdot t_{u})^{a}$ for a constant $a>1$.

4.1 Proof of the Peak-to-Average Lemma

In this subsection we prove our key technical lemma, which is required to complete the proof of Theorem 1.

Lemma 1 (restated).

Let $f:\Sigma^{k}\to\mathbb{R}$ be any real function on the length-$k$ strings over alphabet $\Sigma$, satisfying:

1. (i)

$\sum_{z\in\Sigma^{k}}|f(z)|\leq 1$; and 2. (ii)

$\max_{z\in\Sigma^{k}}|f(z)|\geq\epsilon$**

for some $\epsilon\in[2^{-O(k)},1]$. Then there exists a subset $Y$ of indices, $|Y|\leq O\left(\sqrt{k\cdot\lg 1/\epsilon}\right)$, such that

[TABLE]

The lemma is tight, as shown in Section B of the Appendix. We first provide a high-level overview of the proof of Lemma 1, and then proceed to the formal proof. The first observation is that we may without loss of generality assume that $\Sigma=\{0,1\}$ (intuitively, the larger the alphabet is, the more information we will learn upon revealing the values of $Y$). Assuming $f$ is defined on a boolean hypercube, the high-level intuition for the proof is that we can multiply $f(z)$ by a low-degree polynomial $Q(z)$ that point-wise approximates the $\mathsf{AND}_{k}$ function to within an additive error $\epsilon$, a.k.a., a (variant of the) discrete Chebychev polynomial, which has the effect of preserving the $\ell_{\infty}$ value of $f$ but exponentially “dampening” the magnitude of all remaining values, thereby making the maximum (constant) value of $(f\cdot Q)(z)$ dominate the sum of all remaining values. Since the degree of $Q(z)$ (required to ensure the latter property) is $d:=O(\sqrt{k\lg(1/\epsilon)})$, $Q$ itself can be written as the sum of at most ${k\choose d}=\exp(\tilde{O}(\sqrt{k\cdot\lg 1/\epsilon}))$ monomials, hence one of these monomials (which can be viewed as some specific subset of $\leq d=O(\sqrt{k\lg(1/\epsilon)})$ coordinates) must account for at least $\gtrsim\exp(-\sqrt{k\cdot\lg 1/\epsilon})$ fraction of the total sum $\sum_{z}f(z)\cdot Q(z)$, so fixing this particular monomial’s coordinates (which is the small subset $Y$ we are looking for) to $1$ must contribute the aforementioned quantity to the average of $f$.

4.1.1 The formal proof

As discussed above, a central ingredient of the proof is the existence of low-degree polynomials with “threshold” phenomena, commonly known as Chebychev polynomials. In particular, the following lemma states that there is a low-degree multivariate polynomial that point-wise approximates the AND function on the $k$-dimensional hypercube to within small error (i.e., in the $\ell_{\infty}$ sense). The following lemma, which is translating the quantum algorithm in [BCdWZ99], asserts the existence of such polynomials.

Lemma 2**.**

For any $k$ and $M$ satisfying $2\leq M\leq 2^{O(k)}$, there exists a polynomial $Q=Q_{k,M}(x_{1},\ldots,x_{k})$ such that

1. (i)

$Q$* has total degree $O(\sqrt{k\lg M})$;* 2. (ii)

$|Q(0^{k})|\geq M$; 3. (iii)

$\forall x\in\{0,1\}^{k}\setminus\{0^{k}\},\left|Q(x)\right|\leq 1$; 4. (iv)

The sum of absolute values of all coefficients is at most $\exp(\sqrt{k\lg M})$.

The proof of the lemma can be found in Appendix A. We are now ready to prove the Peak-to-Average Lemma.

Proof of Lemma 1.

We first show that without loss of generality, we may assume that $\Sigma=\{0,1\}$ and $|f(0)|\geq\epsilon$. In general, let $z^{*}\in\Sigma^{k}$ be any point with large absolute $f$-value: $|f(z^{*})|\geq\epsilon$. Define $h:\{0,1\}^{k}\to\mathbb{R}$ as follows:

[TABLE]

It is easy to verify that

[TABLE]

and $|h(0)|=|f(z^{*})|\geq\epsilon$, i.e., $h$ satisfies both conditions in the lemma statement. Moreover, for any subset $Y$ of indices, we have

[TABLE]

Thus, it suffices to prove the lemma assuming $\Sigma=\{0,1\}$ and $|f(0)|\geq\epsilon$.

Let $Q=Q_{k,2/\epsilon}$ be a polynomial with all four properties in Lemma 2 with $M=2/\epsilon$. Since $2^{-O(k)}\leq\epsilon\leq 1$, such polynomial exists and has degree $d\leq O(\sqrt{k\lg M})$. Without loss of generality, we may assume $Q$ is multi-linear.999$Q$ is only evaluated on $\{0,1\}^{k}$, and all four properties are preserved when replacing $x_{i}^{2}$ by $x_{i}$. Thus, let

[TABLE]

By Property (iv), we have

[TABLE]

Now, consider the function

[TABLE]

By the premise of the lemma, we have that $|g(0^{k})|\geq\frac{2}{\epsilon}\cdot\epsilon=2$, but

[TABLE]

by the triangle inequality and $|Q(x)|\leq 1$ for $x\neq 0^{k}$. Therefore, we have

[TABLE]

On the other hand, we have

[TABLE]

By Equation (10) and Equation (9), there must exist some $Y\subseteq[k]$ and $|Y|\leq d$ for which

[TABLE]

Thus, we have

[TABLE]

and $|Y|\leq d=O(\sqrt{k\cdot\lg 1/\epsilon})$, as claimed. ∎

5 Boolean Polynomial Evaluation

In this section, we prove our first concrete lower bound using our new technique. Let $\mathcal{P}$ be the dynamic least-bit polynomial evaluation problem over the Galois field $GF(2^{d})$ (as defined in Section 1.2). Recall that the data structure problem $\mathcal{P}$ is defined as follows: A degree $n\leq 2^{d/4}$ polynomial $P(x)=\sum_{i=0}^{n}a_{i}x^{i}$ over $GF(2^{d})$ is initialized with all $n+1$ coefficients $a_{i}$ being [math]. An update is specified by a tuple $(i,b)$ where $i\in[n+1]$ is an index and $b$ is an element in $GF(2^{d})$. It changes the coefficient $a_{i}$ such that $a_{i}\leftarrow a_{i}+b$ (where addition is over $GF(2^{d})$). A query is specified by an element $y\in GF(2^{d})$ and one must return the least significant bit of $P(y)$. Note that we make no assumptions on the concrete representation of the elements in $GF(2^{d})$, only that the elements are in a bijection with $\{0,1\}^{d}$ so that precisely half of all elements in $GF(2^{d})$ have a [math] as the least significant bit.

We consider the following random sequence of updates $\mathcal{U}=(\mathbf{U}_{\ell},\mathbf{U}_{\ell-1},\ldots,\mathbf{U}_{1})$, where $|\mathbf{U}_{i}|=n_{i}=\beta^{i}$ for some $\beta=(wt_{u})^{\Theta(1)}$. The maintained polynomial $P(x)$ has degree $n=\beta^{\ell}-1$, and thus the number of updates is $\Theta(n)$. The $n_{i}$ updates $\mathbf{U}_{i}$ are $(0,b_{0}),\dots,(n_{i-1},b_{n_{i}-1})$ in that order, where the $b_{j}$’s are chosen independently and uniformly at random from $GF(2^{d})$. The query $q$ is chosen as a uniform random element of $GF(2^{d})$.

Invoking Theorem 1, the existence of a a dynamic data structure for $\mathcal{P}$ with worst case update time $t_{u}$ and expected query time $t_{q}$ under $\mathcal{U}$ implies that either $t_{q}(wt_{u})^{\Theta(1)}>n^{1/2}$ or for some $i\in\{\ell/2,\dots,\ell\}$, we have

[TABLE]

In the first case, we are already done as we have a lower bound of $t_{q}=\Omega(\sqrt{n}/(wt_{u})^{O(1)})$. We thus set out to prove an upper bound on $\overrightarrow{\mathsf{adv}}\left(G^{i}_{\mathcal{P}},\;\mathcal{U},\;n_{i}/(w\cdot t_{u})^{\Theta(1)}\right)$ for any epoch $i\in\{\ell/2,\dots,\ell\}$ (assuming $t_{q}(wt_{u})^{\Theta(1)}\leq n^{1/2}$).

Lemma 3**.**

For any epoch $i\in\{\ell/2,\dots,\ell\}$, we have $\overrightarrow{\mathsf{adv}}\left(G^{i}_{\mathcal{P}},\;\mathcal{U},\;o(n_{i}d)\right)\leq 2^{-d/8}$.

Before proving the lemma, let us use it to derive our lower bound. We see that it must be the case that:

[TABLE]

Thus Theorem 2 follows. Proving Lemma 3 is the focus of Section 5.1.

5.1 Low Advantage on Epochs

Let $i\in\{\ell/2,\dots,\ell\}$. Recall that in the communication game $G^{i}_{\mathcal{P}}$ for the random update sequence $\mathcal{U}$, Alice receives all updates of all epochs and Bob receives all updates of all epochs except epoch $i$. Bob also receives the query $q$. Let $\pi$ be a one-way randomized protocol in which Alice sends $o(n_{i}d)$ bits to Bob, and suppose that $\pi$ achieves an advantage of $\epsilon$ w.r.t $\mathcal{U}$ and $q$. Since the query $q$ and the updates $\mathbf{U}_{i}$ are independent of the updates of epochs $\mathbf{U}_{\ell},\dots,\mathbf{U}_{i+1},\mathbf{U}_{i-1},\dots,\mathbf{U}_{1}$, we can fix the random coins of the protocol and fix the updates of all epochs except epoch $i$, such that for the resulting deterministic protocol $\pi^{*}$ and fixed update sequence $u_{\ell},\dots,u_{i+1},u_{i-1},\dots,u_{1}$ we have that Alice never sends more than $o(n_{i}d)$ bits and $\Pr_{\mathbf{U}_{i},q}[v_{\pi^{*}}=P_{0}(q)]\geq 1/2+\epsilon$. Here $P_{0}$ is the least significant bit of $P(q)$, where $P$ is the polynomial resulting from performing the updates $u_{\ell},\dots,u_{i+1},\mathbf{U}_{i},u_{i-1},\dots,u_{1}$. Recall that the random variable $v_{\pi^{*}}$ is Bob’s output when running the deterministic protocol $\pi^{*}$ on $u_{\ell},\dots,u_{i+1},\mathbf{U}_{i},u_{i-1},\dots,u_{1}$ and $q$.

Let $M_{\pi^{*}}(\mathbf{U}_{i})$ denote the message sent by Alice in procotol $\pi^{*}$ on updates $u_{\ell},\dots,u_{i+1},\mathbf{U}_{i},u_{i-1},\dots,u_{1}$. Then $v_{\pi^{*}}=v_{\pi^{*}}(M_{\pi^{*}}(\mathbf{U}_{i}),q)$ is determined from $M_{\pi^{*}}(\mathbf{U}_{i})$ and $q$ alone (since the updates of other epochs are fixed). For each of the possible messages $m$ of Alice, define the vector $\chi_{m}$ having one coordinate per $x\in GF(2^{d})$. The coordinate $\chi_{m}(x)$ corresponding to some $x$ has the value $-1$ if $v_{\pi^{*}}(m,x)=0$ and it has the value $1$ otherwise. Similarly define for each sequence of updates $u_{i}\in\operatorname{supp}(\mathbf{U}_{i})$ the vector $\psi_{u_{i}}$ having one coordinate $\psi_{u_{i}}(x)$ per $x\in GF(2^{d})$, where the coordinate corresponding to some $x$ takes the value $-1$ if the correct answer to the query $x$ is [math] after the update sequence $u_{\ell},\dots,u_{i+1},u_{i},u_{i-1},\dots,u_{1}$ and taking the value $1$ otherwise. Since $\pi^{*}$ has advantage $\epsilon$ and $q$ is uniform over $GF(2^{d})$, we must have

[TABLE]

This in particular implies that if we take the absolute value of the inner product, we have

[TABLE]

As we will show later, this implies the following:

Lemma 4**.**

There is some $m\in\operatorname{supp}(M_{\pi^{*}}(\mathbf{U}_{i}))$ such that we have both

• •

$\mathop{{}\mathbb{E}}_{\mathbf{U}_{i}}[|\langle\psi_{\mathbf{U}_{i}},\chi_{M_{\pi^{*}}(\mathbf{U}_{i})}\rangle|\mid M_{\pi^{*}}(\mathbf{U}_{i})=m]\geq\epsilon 2^{d}.$**

• •

$\Pr_{\mathbf{U}_{i}}[M_{\pi^{*}}(\mathbf{U}_{i})=m]\geq|\operatorname{supp}(M_{\pi^{*}}(\mathbf{U}_{i}))|^{-1}\epsilon/2.$**

Consider such an $m$ and the corresponding vector $\chi_{m}$. We examine the following $k$’th moment for an even $k$ to be determined:

[TABLE]

Here $\prod_{x\in S}$ is the product over all elements in the tuple $S$ (where some elements may occur multiple times). Now observe that the polynomial $P$ corresponding to an update sequence $u_{\ell},\dots,u_{1}$ can be written as the sum of two polynomials $P=Q+R$, where $Q$ is the polynomial corresponding to performing all updates except those in epoch $i$, and $R$ corresponds to performing only the updates of epoch $i$. Since we fixed all updates outside epoch $i$, the polynomial $Q$ is fixed. The polynomial $R$ on the other hand is precisely uniform random over all degree $n_{i}-1$ polynomials over $GF(2^{d})$. It follows that the evaluations $P(x)$ are $n_{i}$-wise independent, i.e. for any $h\leq n_{i}$ distinct elements $x_{1},\dots,x_{h}\in GF(2^{d})$ and any set of (not necessarily distinct) values $y_{1},\dots,y_{h}\in GF(2^{d})$, we have

[TABLE]

This in particular implies that the entries of $\psi_{\mathbf{U}_{i}}$ are uniform random and $n_{i}$-wise independent. Using this observation, we observe that

[TABLE]

is [math] if $k\leq n_{i}$ and at least one $x$ occurs an odd number of times in $S$. On the other hand, if all $x$ in $S$ occur an even number of times, then both $\mathop{{}\mathbb{E}}_{\mathbf{U}_{i}}\left[\prod_{x\in S}\psi_{\mathbf{U}_{i}}(x)\right]$ and $\prod_{x\in S}\chi_{m}(x)$ are equal to $1$. The number of tuples $S\in GF(2^{d})^{k}$ with all elements occuring an even number of times is at most $\binom{2^{d}+k/2-1}{k/2}k!$. We thus have

[TABLE]

Since $k$ is even, we know that $\langle\psi_{\mathbf{U}_{i}},\chi_{m}\rangle^{k}$ is non-negative. Thus we can insert absolute values:

[TABLE]

Using that $\Pr_{\mathbf{U}_{i}}[M_{\pi^{*}}(\mathbf{U}_{i})=m]\geq|\operatorname{supp}(M_{\pi^{*}}(\mathbf{U}_{i}))|^{-1}\epsilon/2$ by the second proposition of Lemma 4, it further holds that

[TABLE]

By convexity of $x^{k}$, it follows from Jensen’s inequality that

[TABLE]

Taking the $k$’th root, we arrive at

[TABLE]

We thus conclude

[TABLE]

Setting $k=n_{i}$ (or $n_{i}-1$ if $n_{i}$ is odd), we have $k\leq 2^{d/4}$ (since $n_{i}\leq n_{\ell}\leq 2^{d/4}$). For this choice of $k$, the above gives us:

[TABLE]

For $\epsilon\geq 2^{-d/8}$, this gives

[TABLE]

Since we assumed the protocol has Alice sending $o(n_{i}d)$ bits, we conclude that the protocol $\pi$ must have advantage less than $2^{-d/8}$. Since this holds for any protocol, we conclude $\overrightarrow{\mathsf{adv}}\left(G^{i}_{\mathcal{P}},\;\mathcal{U},\;o(n_{i}d)\right)\leq 2^{-d/8}$.

Proof of Lemma 4.

For each $m\in\operatorname{supp}(M_{\pi^{*}}(\mathbf{U}_{i}))$, define $z_{m}$ to take the value $z_{m}:=1/\Pr_{\mathbf{U}_{i}}[M_{\pi^{*}}(\mathbf{U}_{i})=m]$ and $y_{m}:=2^{d}-\mathop{{}\mathbb{E}}_{\mathbf{U}_{i}}[|\langle\psi_{\mathbf{U}_{i}},\chi_{M_{\pi^{*}}(\mathbf{U}_{i})}\rangle|\mid M_{\pi^{*}}(\mathbf{U}_{i})=m]$. Both $z_{m}$ and $y_{m}$ are non-negative for all $m\in\operatorname{supp}(M_{\pi^{*}}(\mathbf{U}_{i}))$. We observe that $\mathop{{}\mathbb{E}}_{\mathbf{U}_{i}}[z_{M_{\pi^{*}}(\mathbf{U}_{i})}]=|\operatorname{supp}(M_{\pi^{*}}(\mathbf{U}_{i}))|$. Secondly, we have

[TABLE]

From Markov’s inequality, we conclude that

[TABLE]

Similarly, we conclude

[TABLE]

By a union bound, we conclude that there is some $m\in\operatorname{supp}(M_{\pi^{*}}(\mathbf{U}_{i}))$ satisfying both:

[TABLE]

and

[TABLE]

6 Lower Bound for Parity Searching in Butterfly Graphs

In this section, we prove a lower bound for a new and dynamic version of Pǎtraşcu’s distance oracles on “Butterfly” graphs, which is the key for the rest of our lower bounds. We start by formally defining the problem, which we call parity searching in Butterfly graphs.

6.1 Parity Searching in Butterfly Graphs

To introduce the problem, we first define the Butterfly graph. The Butterfly graph with degree $B$ and depth $d$ is defined as follows: It has $d+1$ layers, each with $B^{d}$ vertices. The vertices on level [math] are sources, while the ones on level $d$ are sinks. Each vertex except the sinks has out-degree $B$, and each vertex except the sources has in-degree $B$. Let $v_{0},\dots,v_{B^{d}-1}$ denote the nodes at some level of the Butterfly. We think of the nodes as being indexed by vectors in $[B]^{d}$, where the $i$’th coordinate of the vector corresponding to $v_{j}$ equals the $i$’th bit of $j$. With this representation, there are precisely two edges going out of a node $v_{j}$ on level $i$, and these two edges go to the two nodes $w_{j}$ and $w_{k}$ at level $i+1$ such that $k$ differs from $j$ only on the $i$’th bit (and $j=j$). See Figure 2 for a Butterfly graph with degree $2$ and depth $3$.

The Butterfly graph has the property that there is a unique source-sink path for each source-sink pair, corresponding to “morphing” the binary representation of the source index into the sink index, one bit at a time.

For a directed edge $(u,v)$ in the Butterfly graph, let $j$ denote the level of $u$ and $j+1$ the level of $v$. Let $\hat{u}=(\dots,w_{j-1},w_{j},w_{j+1},\dots)\in[B]^{d}$ denote the vector corresponding to $u$ and $\hat{v}=(\dots,w_{j-1},w^{\prime}_{j},w_{j+1},\dots)\in[B]^{d}$ denote the vector corresponding to $v$. The crucial property of the Butterfly graph is that the set of source-sink pairs $(s,t)$ that have their unique path routing through the edge $(u,v)$ are precisely those pairs $(s,t)$ where $s\in(\star,\star,\dots,w_{j},w_{j+1},\dots,w_{d-1})\subseteq[B]^{d}$ and $t\in(w_{0},\dots,w_{j-1},w^{\prime}_{j},\star,\dots)\subseteq[B]^{d}$. See Figure 3.

Pǎtraşcu studied reachability in the Butterfly graph. More precisely, one is given as input a subset of the $n=dB^{d+1}$ possible edges of a Butterfly graph. One must preprocess the edges into a data structure, such that given a source-sink pair $(s,t)$, one can output whether $s$ can reach $t$.

We modify Pǎtraşcu’s reachability problem such that we can use it in reductions to prove new dynamic lower bounds.

Parity Searching in One Butterfly.

First consider the following boolean problem on (a single) Butterfly graph: Each edge $(u,v)$ of the Butterfly is assigned a weight $z^{(u,v)}$ amongst [math] and $1$. A query is defined by a pair $(s,t)\in[B^{d}]\times[B^{d}]$. We think of $s$ as the index of a source node. For $t$, compute the number $\hat{t}=\overleftarrow{t}$ and think of $\hat{t}$ as the index of a sink. The goal is to sum the weights assigned to the set of edges on the unique source-sink path from $s$ to $\hat{t}$. Here $\overleftarrow{x}$ of an integer $x\in[B^{d}]$ is the number obtained by writing $x$ in base $B$ and then reversing the order of digits. That is, if $x=\sum_{j=0}^{d-1}w_{j}B^{j}$ for $w_{j}\in[B]$, then $\overleftarrow{x}=\sum_{j=0}^{d-1}w_{j}B^{d-j-1}$. Note that $\overleftarrow{x}$ preserves the leading [math]’s and thus when reversing $t$ we include potential leading [math]’s in such a way that before reversing, we have precisely $d$ digits. Reversing the digits of $t$ is an idea by Pǎtraşcu. It has the crucial effect that the set of queries $(s,t)$ summing the weight of an edge $(u,v)$ will correspond to a rectangle. More formally, recall that the source-sink pairs $(s^{\prime},t^{\prime})\in[B]^{d}\times[B]^{d}$ that route through an edge $(u,v)$ is precisely the set where $s^{\prime}\in(\star,\star,\dots,w_{j},w_{j+1},\dots,w_{d-1})\subseteq[B]^{d}$ and $t^{\prime}\in(w_{0},\dots,w_{j-1},w^{\prime}_{j},\star,\dots)\subseteq[B]^{d}$. Translating this back to the query pairs $(s,t)\in[B^{d}]\times[B^{d}]$ (where we recall that the sink index is obtained by reversing the digits of $t$), we get that the set of queries that must sum the weight of an edge are those where

[TABLE]

and

[TABLE]

Note that this is a rectangle. This already hints at how we are going to use Butterfly graphs in a reduction to 2D rectangle stabbing.

Denoting by $p(s,t)$ the unique set of edges on the path from the source indexed by $s$ to the sink indexed by $\overleftarrow{t}$, the goal is thus to compute $\oplus_{(u,v)\in p(s,t)}z^{(u,v)}$, where $\oplus$ denotes XOR (parity). See Figure 4.

Parity Searching in Butterfly Graphs.

We extend this problem to a dynamic data structure problem, parity searching in Butterfly graphs, as follows: We have $\ell$ Butterfly graphs $G_{\ell},\dots,G_{1}$ where all $G_{i}$’s have the same degree $B$, but varying depths $d_{\ell},\dots,d_{1}$ such that $d_{i}=8i\lg(t_{u}w)/\lg B$. We will from here and onwards fix $B=(wt_{u})^{8}$ to ensure that depths are integers (they then become $d_{i}=i$). The idea is that we will have an epoch of updates corresponding to each $G_{i}$. Since the number of edges in a Butterfly graph of depth $d_{i}$ is $d_{i}B^{d_{i}+1}$, this means that the epoch sizes go down by a factor $\beta=\Theta(B^{8\lg(t_{u}w)/\lg B})=\Theta((t_{u}w)^{8})$.

Initially all edges of all Butterflies have weight [math]. An update is specified by an index $i\in\{1,\dots,\ell\}$, an edge $(u,v)\in G_{i}$ and a weight $y\in\{0,1\}$. It has the effect of changing the weight of the edge $(u,v)\in G_{i}$ to $z^{(u,v)}\leftarrow y$. For technical reasons (for use in reductions), we require that throughout a sequence of updates, no edge has its weight set more than once.

A query is specified by two indices $s,t\in[B^{d_{\ell}}]$ and the answer to the query is:

[TABLE]

where $p_{i}(s,t)$ is the set of edges on the path from the source $s_{i}=\lfloor s/B^{d_{\ell}-d_{i}}\rfloor$ to the sink $t_{i}=\overleftarrow{\lfloor t/B^{d_{\ell}-d_{i}}\rfloor}$ in $G_{i}$.

To summarize a query in words, the indices $s$ and $t$ are translated to a set of source-sink pairs, one in each $G_{i}$, by integer division with $B^{d_{\ell}-d_{i}}$ and then reversing the bits of the number obtained from $t$. We must then compute the parity of the weights assigned to all the edges along the $\ell$ corresponding paths $p_{1}(s,t),\dots,p_{\ell}(s,t)$. Note that the total input size is $n=\sum_{i=1}^{\ell}d_{i}B^{d_{i}+1}=O(d_{\ell}B^{d_{\ell}+1})$ and that the size of consecutive graphs $G_{i}$ and $G_{i-1}$ differ by a factor $(t_{u}w)^{\Theta(1)}$. See Figure 5 for an example.

6.2 The Lower Bound

Let $\mathcal{P}$ denote the dynamic problem of parity searching in Butterfly graphs. Recall that in $\mathcal{P}$, we have $\ell$ Butterfly graphs $G_{\ell},\dots,G_{1}$ where all $G_{i}$’s have the same degree $B=(wt_{u})^{8}$, but varying depths $d_{\ell},\dots,d_{1}$ such that $d_{i}=8i\lg(t_{u}w)/\lg B=i$.

Hard Distribution.

We consider the following random sequence of updates $\mathcal{U}=(\mathbf{U}_{\ell},\mathbf{U}_{\ell-1},\ldots,\mathbf{U}_{1})$, where $|\mathbf{U}_{i}|=n_{i}=d_{i}B^{d_{i}+1}$. The updates $\mathbf{U}_{i}$ of epoch $i$ assign a uniform random weight amongst $\{0,1\}$ to each edge of the Butterfly $G_{i}$ (which has precisely $d_{i}B^{d_{i}+1}$ edges). The query $q=(s,t)$ has $s$ and $t$ drawn independently and uniformly at random from $[B^{d_{\ell}}]$.

6.2.1 Meta Queries

Let $D$ be a dynamic data structure for parity searching in Butterfly graphs of degree $B$, having worst case update time $t_{u}$ and expected query time $t_{q}$ under $\mathcal{U}$. We cannot apply Theorem 1 directly to this problem by proving a strong lower bound for the possible advantage on epochs $i\in[\ell/2,\ell]$. In fact, there is a randomized one-way protocol achieving advantage $1/\mathrm{poly}(\lg n)$ with $n_{i}/\mathrm{poly}(\lg n)$ communication for any epoch $i$. To get the lower bound we are after, we need to show that with $n_{i}/\mathrm{poly}(\lg n)$ communication, the best achievable advantage is only $1/\mathrm{poly}(n)$. To ensure the latter, we need to perform certain technical manipulations on queries of $\mathcal{P}$ in our simulation. We do this as follows:

First, by arguments similar to those in the proof of Theorem 1, there must be an epoch $i\in[\ell/2,\ell]$ such that

[TABLE]

Recall that $T^{i}_{q}$ is the number of cells associated to epoch $i$ which is probed by $D$ when answering $q$ (see Section 4). Fix such an epoch $i$.

Recall that a query in $\mathcal{Q}$ is specified by a tuple $(s,t)$ with $s,t\in[B^{d_{\ell}}]$. Each such tuple corresponds to a source-sink path by integer division with $B^{d_{\ell}-d_{i}}$ and reversing the digits of the number obtained from $t$. Hence there are exactly $B^{d_{\ell}-d_{i}}$ values of $s$ that specify the same source. Likewise for the sinks.

From the set of source-sink paths, we define a collection of meta queries $\mathcal{Q}^{*}$. For each level $j$ of the Butterfly graph $G_{i}$, recall that edges go between vertices whose base-$B$ vector differ only in the $j$’th coordinate. We thus group vertices in level $j$ and $j+1$ into $B^{d_{i}-1}$ chunks. Each chunk consists of all vertices in level $j$ and $j+1$ whose corresponding vectors agree in all but the $j$’th coordinate, i.e. a chunk consists of all vertices with the corresponding vector being $(w_{0},\dots,w_{j-1},\star,w_{j+1},\dots,w_{d_{i}-1})$. Summed over all $d_{i}$ levels with outgoing edges, we have $d_{i}B^{d_{i}-1}$ chunks. Now consider assigning a permutation on $B$ elements to each chunk. Using $\pi_{j}^{(w_{0},\dots,w_{j-1},\star,w_{j+1},\dots,w_{d_{i}-1})}$ to denote the permutation at level $j$ corresponding to vertices in level $j$ and $j+1$ with vectors of the form $(w_{0},\dots,w_{j-1},\star,w_{j+1},\dots,w_{d_{i}-1})$, such an assignment of permutations to all chunks now yield $B^{d_{i}}$ unique source-sink pairs as follows: For each source $s^{\prime}=(w_{0},\dots,w_{d_{i}-1})$, trace a path as follows: Start by going to the level $1$ vertex with vector $(w_{0}^{\prime}=\pi_{0}^{(\star,w_{1}\dots,w_{d_{i}-1})}(w_{0}),w_{1},\dots,w_{d_{i}-1})$. Then to the level $2$ vertex with vector $(w_{0}^{\prime},w_{1}^{\prime}=\pi_{0}^{(w^{\prime}_{0},\star,w_{2}\dots,w_{d_{i}-1})}(w_{1}),\dots,w_{d_{i}-1})$ and so forth until we reach a sink $t^{\prime}=(w_{0}^{\prime},\dots,w_{d_{i}-1}^{\prime})$. Since we use a permutation in each chunk, the set of $B^{d_{i}}$ constructed source-sink pairs have the property that exactly one path passes through each vertex at each level. See Figure 6.

Now consider such a collection of $B^{d_{i}}$ source-sink pairs $(0,t_{1}),\dots,(B^{d_{i}}-1,t_{B^{d_{i}}-1})$ (recall there is precisely one query per source and one per sink). We create a number of meta queries for each such collection of $B^{d_{i}}$ source-sink pairs. The set of meta queries corresponding to source-sink pairs $(0,t_{0}),\dots,(B^{d_{i}}-1,t_{B^{d_{i}}-1})$ is precisely the set of all query lists $S=(s^{\prime}_{0},t^{\prime}_{0}),\dots,(s^{\prime}_{B^{d_{i}}-1},t^{\prime}_{B^{d_{i}}-1})$ with $\lfloor s^{\prime}_{j}/B^{d_{\ell}-d_{i}}\rfloor=j$ and $\overleftarrow{\lfloor t^{\prime}_{j}/B^{d_{\ell}-d_{i}}\rfloor}=t_{j}$ for all $j=0,\dots,B^{d_{i}}-1$. The meta query $q^{S}$ then has the answer

[TABLE]

Said in words, the meta query $q^{S}$ asks to compute the XOR of the answer to all the queries in $S$. Note that all paths involved in the XORs are disjoint in $G_{i}$, and thus the weight of any edge in $G_{i}$ is included at most once in this sum. We use $\mathcal{Q}^{*}$ to denote the collection of all meta queries.

Now the idea is that we can use $D$ to answer such meta queries efficiently. More specifically, consider running the same distribution $\mathcal{U}$ of updates, but instead of drawing the query tuple $q$ as above, we instead draw a uniform random meta query $q^{S}$ and ask to output its answer. Call the resulting dynamic data structure problem $\mathcal{P}^{*}_{i}$. We can use the data structure $D$ to obtain an efficient solution for this problem: When receiving the meta query $q^{S}$, we simply run the query algorithm for each $(s_{j},t_{j})\in S$ and compute the resulting XOR of query answers. Clearly this gives the correct result. Now the crucial observation is that if we draw a uniform random query $(s_{j},t_{j})$ from $q^{S}$, then the distribution of that query is still uniform over all queries to the original problem, i.e. the distribution of $(s_{j},t_{j})$ is simply a uniform random tuple in $[B^{d_{\ell}}]\times[B^{d_{\ell}}]$. Thus by linearity of expectation, we have:

[TABLE]

Here $T^{i}_{q^{S}}$ is the number of cells associated to epoch $i$ that is probed when answering $q^{S}$ in the above manner. We also have:

[TABLE]

We now wish to invoke Theorem 6. The theorem requires $t_{q}(wt_{u})^{a+1}\leq n_{i}$ for a constant $a>1$. The average expected query time for a meta query was $B^{d_{i}}t_{q}$, thus we must have $B^{d_{i}}t_{q}(wt_{u})^{a+1}\leq n_{i}$ for a constant $a>1$. Since $n_{i}=d_{i}B^{d_{i}+1}$, we see that it suffices to have $B\geq t_{q}(wt_{u})^{a+1}$. We can assume $t_{q}\leq\lg^{2}n$, as otherwise we have already finished our proof. Therefore, we see that any $B=\Omega((wt_{u})^{a+3})$ suffices (as $w=\Omega(\lg n)$). We chose $B=(wt_{u})^{8}$, so we can apply the theorem with any $a\leq 5$. Furthermore, the epoch sizes go down by a factor $\beta=\Theta((wt_{u})^{8})$ which also satisfies the requirements of the theorem for any choice of $a\leq 8$.

We can thus invoke Theorem 6, with $a=2$, to conclude that

[TABLE]

The next section therefore aims to bound the best achievable advantage for this $G^{i}_{\mathcal{P}^{*}_{i}}$ problem.

Before we proceed, a few remarks are in order. As discussed earlier, the base problem of answering just one source-sink pair admits a too efficient communication protocol. As we shall see in the following subsection, forcing the data structure to answer multiple structured queries on the same input (i.e., meta queries) rules out such efficient protocols. Unfortunately, the number of queries we need in a meta query depends on the epoch size. Therefore, we had to first zoom in on an epoch $i$ for which the data structure is efficient, and then define the meta queries after having chosen the epoch. This is also the reason why we needed the more specific Theorem 6 rather than Theorem 1.

The next subsection proves the following result:

Lemma 5**.**

For any epoch $i\in[\ell/2,\ell]$, we have $\overrightarrow{\mathsf{adv}}\left(G^{i}_{\mathcal{P}^{*}_{i}},\;\mathcal{U},\;o(n_{i})\right)\leq 2^{-(d_{i}B^{d_{i}})/8}$.

Let us first see why this lemma implies the desired lower bound. We see that it must be the case that

[TABLE]

But $d_{i}\geq d_{\ell}/2$ for any $i\geq\ell/2$ and $d_{\ell}=\Omega(\lg_{B}n)$, thus we conclude that

[TABLE]

We have set $B=(wt_{u})^{8}$, thus the lower bound becomes:

[TABLE]

This completes the proof of Theorem 3. The next section proves the necessary Lemma 5.

6.2.2 Low Advantage on Epochs

Let $i\in[\ell/2,\ell]$. In the communication game $G^{i}_{\mathcal{P}^{*}_{i}}$ for the random update sequence $\mathcal{U}$, Alice receives all updates of all epochs and Bob receives all updates of all epochs except $i$. Bob also receives a meta query $q^{S}$. Let $\pi$ be a one-way randomized protocol in which Alice sends $o(n_{i})$ bits to Bob, and suppose that $\pi$ achieves an advantage of $\epsilon$ w.r.t. $\mathbf{U}$ and $q^{S}$. Since the query $q^{S}$ and the updates $\mathbf{U}_{i}$ are independent of the updates of epochs $\mathbf{U}_{\ell},\cdots,\mathbf{U}_{i+1},\mathbf{U}_{i-1},\dots,\mathbf{U}_{1}$, we can fix the random coins of the protocol and fix the updates of all epochs except epoch $i$, such that for the resulting deterministic protocol $\pi^{*}$ and fixed update sequences $u_{\ell},\dots,u_{i+1},u_{i-1},\dots,u_{1}$ we have that Alice never sends more than $o(n_{i})$ bits and $\Pr_{\mathbf{U}_{i},q^{S}}[v_{\pi^{*}}=\mathcal{A}(q^{S})]\geq 1/2+\epsilon$. Here $\mathcal{A}(q^{S})$ is the answer to query $q^{S}$ on updates $u_{\ell},\dots,u_{i+1},\mathbf{U}_{i},u_{i-1},\dots,u_{1}$ and query $q^{S}$. The random variable $v_{\pi^{*}}$ is Bob’s output when running the deterministic protocol $\pi^{*}$ on $u_{\ell},\dots,u_{i+1},\mathbf{U}_{i},u_{i-1},\dots,u_{1}$ and query $q^{S}$.

Let $M_{\pi^{*}}(\mathbf{U}_{i})$ be the message sent by Alice in procotol $\pi^{*}$ on updates $u_{\ell},\dots,u_{i+1},\mathbf{U}_{i},u_{i-1},\dots,u_{1}$. Then $v_{\pi^{*}}=v_{\pi^{*}}(M_{\pi^{*}}(\mathbf{U}_{i}),q^{S})$ is determined from $M_{\pi^{*}}(\mathbf{U}_{i})$ and $q^{S}$ alone (since the updates of other epochs are fixed). For each of the possible messages $m$ of Alice, define the vector $\chi_{m}$ having one coordinate per $q^{S}\in\mathcal{Q}^{*}$. The coordinate $\chi_{m}(x)$ corresponding to some $q^{S}$ has the value $-1$ if $v_{\pi^{*}}(m,q^{S})=0$ and it has the value $1$ otherwise. Similarly define for each sequence of updates $u_{i}\in\operatorname{supp}(\mathbf{U}_{i})$ the vector $\psi_{u_{i}}$ having one coordinate $\psi_{u_{i}}(q^{S})$ per $q^{S}\in\mathcal{Q}^{*}$, where the coordinate corresponding to some $x$ takes the value $-1$ if the correct answer to the query $q^{S}$ is [math] after the update sequence $u_{\ell},\dots,u_{i+1},u_{i},u_{i-1},\dots,u_{1}$ and taking the value $1$ otherwise. Since $\pi^{*}$ has advantage $\epsilon$ and $q^{S}$ is uniform in $\mathcal{Q}^{*}$, we must have

[TABLE]

This in particular implies that if we take the absolute value of the inner product, we have

[TABLE]

By arguments identical to those in the proof of Lemma 4, we conclude that there must be some $m\in\operatorname{supp}(M_{\pi^{*}}(\mathbf{U}_{i}))$ such that we have both

• •

$\mathop{{}\mathbb{E}}_{\mathbf{U}_{i}}[|\langle\psi_{\mathbf{U}_{i}},\chi_{M_{\pi^{*}}(\mathbf{U}_{i})}\rangle|\mid M_{\pi^{*}}(\mathbf{U}_{i})=m]\geq\epsilon|\mathcal{Q}^{*}|.$

• •

$\Pr_{\mathbf{U}_{i}}[M_{\pi^{*}}(\mathbf{U}_{i})=m]\geq|\operatorname{supp}(M_{\pi^{*}}(\mathbf{U}_{i}))|^{-1}\epsilon/2.$

Consider such an $m$ and the corresponding vector $\chi_{m}$. We examine the following expectation for an even integer $k$ to be determined:

[TABLE]

Now recall that $\mathbf{U}_{i}$ assigns a uniform random and independently chosen weight in $\{0,1\}$ to each edge of $G_{i}$. For an edge $(u,v)$, let $y^{(u,v)}$ take the value $1$ if $z^{(u,v)}=0$ and let it take the value $-1$ otherwise. Then

[TABLE]

It follows that if there is even a single edge $(u,v)$ in $G_{i}$, such that $(u,v)$ occurs an odd number of times when summed over all $p_{i}(s^{\prime}_{j},t^{\prime}_{j})$ in all $q^{S}$ in $T$, then $\mathop{{}\mathbb{E}}_{\mathbf{U}_{i}}\left[\prod_{q^{S}\in T}\psi_{\mathbf{U}_{i}}(q^{S})\right]=0$. If all edges in $G_{i}$ occur an even number of times, then $\mathop{{}\mathbb{E}}_{\mathbf{U}_{i}}\left[\prod_{q^{S}\in T}\psi_{\mathbf{U}_{i}}(q^{S})\right]\in\{-1,+1\}$, depending on the (fixed) weights assigned to edges in epochs different from $i$. Denoting by $\Gamma$ the number of sets $T\in(\mathcal{Q}^{*})^{k}$ such that all edges in $G_{i}$ occur an even number of times in the corresponding source-sink paths, we conclude that

[TABLE]

Since we assume $k$ is even, we may insert absolute values:

[TABLE]

To bound $\Gamma$, consider drawing $k$ meta queries $q^{S_{1}},q^{S_{2}},\dots,q^{S_{k}}$ independently and uniformly at random. We wish to bound the probability that every edge in $G_{i}$ is included an even number of times when summed over the $k$ meta queries $q^{S_{1}},\dots,q^{S_{k}}$. For this, zoom in on a chunk of the butterfly graph between some levels $j$ and $j+1$. Each $q^{S_{h}}$ assigns a uniform random permutation $\pi_{h}$ on $B$ elements to the chunk. That each edge occurs an even number of times in the chunk is equivalent to every pair of indices $x,y\in[B]$ satisfying that there is an even number of permutations amongst $\pi_{1},\dots,\pi_{k}$ for which $\pi_{h}(x)=y$. Call this event $E$. To bound the probability of $E$, first observe that $\pi_{1},\dots,\pi_{k}$ is uniform random amongst $(B!)^{k}$ possible lists of permutations. We wish to bound the number of such lists $\rho_{1},\dots,\rho_{k}$ that satisfy

For every pair $(x,y)\in[B]\times[B]$, there is an even number of $\rho_{h}$ for which $\rho_{h}(x)=y$.

We upper bound the number of such lists $\rho_{1},\dots,\rho_{k}$ via an encoding argument. Let $\rho_{1},\dots,\rho_{k}$ satisfy 1. We can encode $\rho_{1},\dots,\rho_{k}$ as follows: First specify for every pair $x,y$ how many $\rho_{h}$ that satisfy $\rho_{h}(x)=y$. Letting $\Delta(x,y)$ denote this number for a pair $(x,y)$, we encode the $\Delta(x,y)$’s efficiently as follows: First divide each $\Delta(x,y)$ by $2$. The resulting values are still integer since each $\Delta(x,y)$ is even by assumption. Next observe that $\sum_{(x,y)}\Delta(x,y)/2=kB/2$ as each $\rho_{h}$ adds $B$ to $\sum_{(x,y)}\Delta(x,y)$. Thus we need to specify a sequence of $B^{2}$ non-negative integers that sum to $kB/2$. It is well known that the number of such integer sequences is $\binom{B^{2}+kB/2-1}{kB/2}$, thus we can specify all $\Delta(x,y)$’s using a total of $\lg\binom{B^{2}+kB/2-1}{kB/2}$ bits. Finally, for each pair $(x,y)$ in lexicographic order (i.e. first ordered by $x$, then by $y$), if $\Delta(x,y)>0$, append $\Delta(x,y)\lg k$ bits to the encoding, specifying the set of $\Delta(x,y)$ indices $h$ amongst $\{1,\dots,k\}$ that have $\rho_{h}(x)=y$. Clearly the set $\rho_{1},\dots,\rho_{k}$ can be recovered from this encoding. The number of bits used is upper bounded by $\lg\binom{B^{2}+kB/2-1}{kB/2}+kB\lg k$, meaning that the number of distinct $\rho_{1},\dots,\rho_{k}$ is upper bounded by $\binom{B^{2}+kB/2-1}{kB/2}k^{kB}$. We therefore conclude that

[TABLE]

We now fix $k=B/16$ (we assume $k$ is a even and remark that this can always be achieved by blowing up $w$ and $t_{u}$ by constant factors to ensure that $B=(wt_{u})^{8}$ is a power of $2$). For this value of $k$, the above is bounded by:

[TABLE]

Next recall that the permutations assigned to the chunks are independent. Using the fact that there are $d_{i}B^{d_{i}-1}$ chunks, we conclude that the probability that all edges occur an even number of times in $q^{S_{1}},q^{S_{2}},\dots,q^{S_{k}}$ is less than $2^{-(B^{2}/32)d_{i}B^{d_{i}-1}}$. Therefore, we get that

[TABLE]

Using that $\Pr_{\mathbf{U}_{i}}[M_{\pi^{*}}(\mathbf{U}_{i})=m]\geq|\operatorname{supp}(M_{\pi^{*}}(\mathbf{U}_{i}))|^{-1}\epsilon/2$, we conclude that

[TABLE]

By convexity of $x^{k}$ and Jensen’s inequality, we have

[TABLE]

Taking the $k$’th root, we get

[TABLE]

Using that $\mathop{{}\mathbb{E}}_{\mathbf{U}_{i}}[|\langle\psi_{\mathbf{U}_{i}},\chi_{M_{\pi^{*}}(\mathbf{U}_{i})}\rangle|\mid M_{\pi^{*}}(\mathbf{U}_{i})=m]\geq\epsilon|\mathcal{Q}^{*}|$, we conclude that

[TABLE]

Assuming $\epsilon\geq(1/2)^{(d_{i}B^{d_{i}+1})/(128k)}$, we must have:

[TABLE]

Taking logs, we conclude that

[TABLE]

The contrapositive says that assuming $\lg_{2}\left(|\operatorname{supp}(M_{\pi^{*}}(\mathbf{U}_{i}))|\right)=o(n_{i})$, we must have $\epsilon\leq 2^{-(d_{i}B^{d_{i}+1})/(128k)}$. Inserting $k=B/16$, this becomes $\epsilon\leq 2^{-(d_{i}B^{d_{i}})/8}$.

7 Reductions

This section presents the reductions used to prove the various lower bounds discussed in Section 1.2.

7.1 2D Range Parity and 2D Rectangle Stabbing

As argued in Section 1.2, a folklore reduction shows that 2D range counting and 2D rectangle stabbing are equivalent problems. The same goes for 2D range parity and 2D rectangle parity, where 2D rectangle parity is the version of rectangle stabbing where we need only return the parity of the number of rectangles containing the query point $q$. Thus we start by giving a reduction from parity searching in Butterfly graphs to 2D rectangle parity.

Recall that in parity searching in Butterfly graphs (see Section 6.2), we have $\ell$ Butterfly graphs $G_{\ell},\dots,G_{1}$ where all $G_{i}$’s have the same degree $B$, but varying depths $d_{\ell},\dots,d_{1}$. Initially all edges of all Butterflies have weight [math]. An update is specified by an index $i\in\{1,\dots,\ell\}$, an edge $(u,v)\in G_{i}$ and a weight $y\in\{0,1\}$. It has the effect of changing the weight of the edge $(u,v)\in G_{i}$ to $z^{(u,v)}\leftarrow y$. A query is specified by two indices $s,t\in[B^{d_{\ell}}]$ and the answer to the query is:

[TABLE]

where $p_{i}(s,t)$ is the set of edges on the path from the source $s_{i}=\lfloor s/B^{d_{\ell}-d_{i}}\rfloor$ to the sink $t_{i}=\overleftarrow{\lfloor t/B^{d_{\ell}-d_{i}}\rfloor}$ in $G_{i}$.

We want to show that we can solve this problem using a dynamic data structure for 2D rectangle parity. To do this, we associate a rectangle with each edge $(u,v)$ in some $G_{i}$. Let $j$ denote the level of $u$ and $j+1$ the level of $v$. Recall the notation from Section 1.2 and let $\hat{u}=(\dots,w_{j-1},w_{j},w_{j+1},\dots)\in[B]^{d_{i}}$ denote the vector corresponding to $u$ and $\hat{v}=(\dots,w_{j-1},w^{\prime}_{j},w_{j+1},\dots)\in[B]^{d_{i}}$ denote the vector corresponding to $v$. The crucial property of the Butterfly graph is that the set of source-sink pairs $(s_{i},t_{i})$ in $G_{i}$ that have their unique path routing through the edge $(u,v)$ are precisely those pairs $(s_{i},t_{i})$ where $s_{i}\in(\star,\dots,\star,w_{j},w_{j+1},\dots,w_{d_{i}-1})\subseteq[B]^{d_{i}}$ and $t_{i}\in(w_{0},\dots,w_{j-1},w^{\prime}_{j},\star,\dots)\subseteq[B]^{d_{i}}$.

Now recall that from a query $(s,t)\in[B^{d_{\ell}}]\times[B^{d_{\ell}}]$ to parity searching in Butterfly graphs, we must sum the weights along the paths between the source-sink pairs $s_{i}=\lfloor s/B^{d_{\ell}-d_{i}}\rfloor$ and $t_{i}=\overleftarrow{\lfloor t/B^{d_{\ell}-d_{i}}\rfloor}$ in $G_{i}$ for each $i=1,\dots,\ell$. If we write $(s,t)\in[B^{d_{\ell}}]\times[B^{d_{\ell}}]$ as the vectors $s=(w^{s}_{0},\dots,w^{s}_{d_{\ell}-1})$ and $t=(w^{t}_{0},\dots,w^{t}_{d_{\ell}-1})$ where $w_{0}^{s}$ and $w^{t}_{0}$ are the least significant digits of $s$ and $t$ in base $B$, then we have that $s_{i}=(w^{s}_{d_{\ell}-d_{i}},\dots,w^{s}_{d_{\ell}-1})$ and $t_{i}=(w^{t}_{d_{\ell}-1},\dots,w^{t}_{d_{\ell}-d_{i}})$. We conclude that the weight of the edge $(u,v)$ in $G_{i}$ must be counted iff

• •

$w^{s}_{k+(d_{\ell}-d_{i})}=w_{k}$ for all $k=j,\dots,d_{i}-1$.

• •

$w^{t}_{d_{\ell}-1-k}=w_{k}$ for $k=0,\dots,j-1$.

• •

$w^{t}_{d_{\ell}-j-1}=w^{\prime}_{j}$.

But $s$ and $t$ are integers in $[B^{d_{\ell}}]$ and the above requirements are thus captured precisely by:

• •

$s\in\left[\sum_{k=j}^{d_{i}-1}w_{k}B^{k+(d_{\ell}-d_{i})},\sum_{k=j}^{d_{i}-1}w_{k}B^{k+(d_{\ell}-d_{i})}+\sum_{k=0}^{j-1+(d_{\ell}-d_{i})}(B-1)B^{k}\right]$.

• •

$t\in\left[w^{\prime}_{j}B^{d_{\ell}-j-1}+\sum_{k=0}^{j-1}w_{k}B^{d_{\ell}-1-k},w^{\prime}_{j}B^{d_{\ell}-j-1}+\sum_{k=0}^{j-1}w_{k}B^{d_{\ell}-1-k}+\sum_{k=0}^{d_{\ell}-j-2}(B-1)B^{k}\right]$.

This is equivalent to the point $(s,t)$ being inside a 2D rectangle that depends only on $(u,v)$ and $i$. We denote this rectangle by $R^{(u,v)}_{i}$. Our reduction now goes as follows: On an update setting the weight of an edge $(u,v)$ in $G_{i}$ to [math], we do nothing. If the update sets the weight to $1$, we insert the rectangle $R^{(u,v)}_{i}$. To answer a query $(s,t)$ to parity searching in Butterfly graphs, we simply ask the 2D rectangle parity query $(s,t)$. Correctness follows immediately by the above arguments and the fact that the definition of parity searching in Butterfly graphs from Section 1.2 requires that any edge has its weight set at most once. To the worried reader, note that translating an edge $(u,v)$ in $G_{i}$ to $R^{(u,v)}_{i}$ does not require any memory lookups, it is purely computational. Thus in the cell probe model, the translation is free of charge.

For completeness, we also sketch the reduction from 2D rectangle parity to 2D range parity: On an insertion of a rectangle $R=[x_{1},x_{2}]\times[y_{1},y_{2}]$, we insert four points in the 2D range parity data structure, one roughly at each corner of $R$. The observation is that a point $q$ is inside $R$ iff it dominates only $x_{1}$ (if we ignore $q$ lying on one of the sides of $R$). Additionally, if $q$ is not inside $R$, then it dominates an even number of corners of $R$. Hence the parity of the number of rectangles stabbed is the same as the parity of the number of points dominated if we replace each rectangle with a point at each corner. The only minor issue is when $q$ lies on one of the sides of $R$. We give a full reduction also handling this case in the following:

Recall we require coordinates to be integer. From the rectangle $R$, we insert the four points $p_{1}=(2x_{1},2y_{1}),p_{2}=(2x_{1},2y_{2}+1),p_{3}=(2x_{2}+1,2y_{1}),p_{4}=(2x_{2}+1,2y_{2}+1)$. For a 2D rectangle parity query point $q=(x,y)$, we ask the 2D parity counting query $(2x,2y)$. The factors of $2$ and the additive $+1$ is used to handle the “on one of the sides” case. The crucial observation is that

[TABLE]

Similarly

[TABLE]

Also, if $x>x_{2}$ then $2x>2x_{2}+1$ and if $y>y_{2}$ then $2y>2y_{2}+1$ because all coordinates were integer. It follows that if $q$ is inside the rectangle $R$, then $(2x,2y)$ dominates precisely $p_{1}$. If $x<x_{1}$ or $y<y_{1}$, then $(2x,2y)$ dominates none of the points $p_{1},p_{2},p_{3}$ and $p_{4}$. If $x>x_{2}$ but $y_{1}\leq y\leq y_{2}$, then $(2x,2y)$ dominates precisely $p_{1}$ and $p_{3}$. If $x_{1}\leq x\leq x_{2}$ and $y>y_{2}$, then $(2x,2y)$ dominates precisely $p_{1}$ and $p_{2}$. Finally, if $x>x_{2}$ and $y>y_{2}$, then $(2x,2y)$ dominates all the points $p_{1},p_{2},p_{3}$ and $p_{4}$. We conclude that $(2x,2y)$ dominates an odd number of points amongst $p_{1},p_{2},p_{3}$ and $p_{4}$ iff $q$ is inside the rectangle $R$. Hence the result of the 2D range parity query is the same as the result of the 2D rectangle parity query. This concludes the proof of Theorem 4.

7.2 Range Selection and Range Median

In this section, we show that a data structure for dynamic range selection solves parity searching in Butterfly graphs. Recall that in range selection, we are given an array $A=\{A[0],\dots,A[n-1]\}$ of integers, initially all [math]. A query is triple $(i,j,k)$. The goal is to return the index of the $k$’th smallest element in $\{A[i],\dots,A[j]\}$, breaking ties arbitrarily. We prove our lower bound for the special case of prefix selection in which we force $i=0$ and where we are required to return only whether the $k$’th smallest element is stored in an evenly indexed position or an odd one.

We start by describing our reduction in the setting where we need to return the whole index of the $k$’th smallest element, not just the parity. At the end, we argue that all we really need is the parity of the position. Our reduction starts by re-executing the reduction to 2D rectangle parity from Section 7.1. That is, each edge $(u,v)$ of a Butterfly graph $G_{i}$ is mapped to a rectangle $R^{(u,v)}_{i}$ such that a query $(s,t)$ to parity searching in Butterfly graphs must sum the weight $z^{(u,v)}$ of $(u,v)$ iff the point $(s,t)$ is inside the rectangle $R^{(u,v)}_{i}$. We will also use the fact that the proof of our lower bound for parity searching in Butterfly graphs is for a distribution where every edge has its weight set before we query. Thus for our reduction, it suffices to show that we can solve the parity searching query $(s,t)$ after a sequence of updates that have set the weight of every edge of every $G_{i}$. Finally, we also exploit that the rectangles obtained by reduction from the Butterfly graphs have the property that $R^{(u,v)}_{i}$ and $R^{(x,y)}_{j}$ are disjoint if $i=j$ and the edges $(u,v)$ and $(x,y)$ are at the same depth of the Butterfly graph $G_{i}$. This implies that any point is contained in no more than $\Delta:=\sum_{i=1}^{\ell}d_{i}=O(\lg^{2}n)$ rectangles, one for each layer of each Butterfly ($d_{i}$ is the depth of Butterfly $G_{i}$).

The instance we create is the concatenation of two arrays $B$ and $A$, i.e. $C:=B\circ A$. By concatenation, we simply mean that $C[i]:=B[i]$ in case $i\in[|B|]$ and otherwise $C[i]:=A[i-|B|]$.

The $B$ array.

The array $B$ is quite simple. If $n$ denotes the total number of rectangles, then $B$ has $4n\Delta$ entries. We think of $B$ as being partitioned into batches of $(\Delta+1)$ entries, where the $j$’th batch consists of entries $B[(\Delta+1)j],B[(\Delta+1)j+1],\dots,B[(\Delta+1)(j+1)-1]$. We set the entries of $B$ during epoch $\ell$ (the biggest epoch), such that each update of epoch $\ell$ sets $O(\Delta)$ entries of $B$. This means that the worst case update time of the data structure goes up by a factor $O(\Delta)=O(\lg^{2}n)$. The values assigned to the entries of $B$ are as follows: In the $j$’th batch, we set the values to $B[(\Delta+1)j+i]\leftarrow(\Delta+2)j+i+1$ for $i=0,\dots,\Delta$.

The $A$ array.

From each rectangle $R^{(u,v)}_{i}=[x_{1},x_{2}]\times[y_{1},y_{2}]$, we define the four points $p_{1}=(2x_{1},2y_{1}),p_{2}=(2x_{1},2y_{2}+1),p_{3}=(2x_{2}+1,2y_{1})$ and $p_{4}=(2x_{2}+1,2y_{2}+1)$. Then by the arguments of the previous section, a query point $(s,t)$ is inside $R^{(u,v)}_{i}$ iff $(2s,2t)$ dominates only $p_{1}$. Otherwise, $(2s,2t)$ dominates either precisely $\{p_{1},p_{2}\}$, $\{p_{1},p_{3}\}$ or $\{p_{1},p_{2},p_{3},p_{4}\}$. Let $P$ be the collection of all points $p_{1},p_{2},p_{3},p_{4}$ defined from the rectangles $R^{(u,v)}_{i}$ for every $i=1,\dots,\ell$ and every edge $(u,v)\in G_{i}$. Note that $P$ may contain duplicates. We keep the duplicates, so $P$ is multi-set and $|P|=4n$. Also observe that the collection $P$ is fixed and independent of any weights assigned to edges, it is merely a predefined set of points. For each point $p\in P$, define $\operatorname{rank}_{x}(p)$ to be the rank of $p.x$ amongst all $x$-coordinates of points in $P$. Since several points may have the same $x$-coordinate, we break ties in some arbitrary but fixed manner such that all ranks are unique integers from [math] to $|P|-1$. Our maintained array $A$ has one entry for each point $p\in P$, namely entry $A[\operatorname{rank}_{x}(p)]$. We similarly define $\operatorname{rank}_{y}(p)$ for each $p$, such that $\operatorname{rank}_{y}(p)$ is the rank of $p.y$ amongst all $y$-coordinates of points in $P$. Again, we break ties in some arbitrary but fixed manner. Note that $\operatorname{rank}_{x}(p)$ and $\operatorname{rank}_{y}(p)$ are fixed and independent of any weights assigned to edges.

Initially all entries of $A$ are [math]. On an update setting the weight of $R^{(u,v)}_{i}$ to $1$, we do the updates:

• •

$A[\operatorname{rank}_{x}(p_{1})]\leftarrow\operatorname{rank}_{y}(p_{1})(\Delta+2)$.

• •

$A[\operatorname{rank}_{x}(p_{2})]\leftarrow|P|(\Delta+2)$.

• •

$A[\operatorname{rank}_{x}(p_{3})]\leftarrow|P|(\Delta+2)$.

• •

$A[\operatorname{rank}_{x}(p_{4})]\leftarrow\operatorname{rank}_{y}(p_{4})(\Delta+2)$.

On an update setting the weight of $R^{(u,v)}_{i}$ to [math], we instead do the updates:

• •

$A[\operatorname{rank}_{x}(p_{1})]\leftarrow|P|(\Delta+2)$.

• •

$A[\operatorname{rank}_{x}(p_{2})]\leftarrow\operatorname{rank}_{y}(p_{2})(\Delta+2)$.

• •

$A[\operatorname{rank}_{x}(p_{3})]\leftarrow\operatorname{rank}_{y}(p_{3})(\Delta+2)$.

• •

$A[\operatorname{rank}_{x}(p_{4})]\leftarrow|P|(\Delta+2)$.

Conceptually, think of the above choices as inserting the points $p_{1}$ and $p_{4}$ on a weight of $1$, and inserting the points $p_{2}$ and $p_{3}$ on a weight of [math]. Setting the remaining entries to $|P|(\Delta+2)$ can be thought of as disregarding these entries in selection queries.

Answering the Query.

On a query $(s,t)$, we compute the index $j\in[|P|]$ such that all points $p$ in $P$ with $p.x\leq 2s$ have $\operatorname{rank}_{x}(p)\leq j$ and all points $p$ in $P$ with $p.x>2s$ have $\operatorname{rank}_{x}(p)>j$. Note that this index is completely determined from the query $(s,t)$ and does not depend on any weights assigned. Since computation is free in the cell probe model, computing $j$ is free of charge. Similarly, we can compute the index $h\in[|P|]$ such that all points $p\in P$ with $p.y\leq 2t$ have $\operatorname{rank}_{y}(p)\leq h$ and all points $p\in P$ with $p.y>2t$ have $\operatorname{rank}_{y}(p)>h$.

The crucial observation is that if we disregard the rectangles $R^{(u,v)}_{i}$ containing $(s,t)$ (i.e. disregard the corresponding entries in $A$), then we know exactly what value of $k$ that makes the query $(0,j+|B|,k)$ return the index $r:=h(\Delta+1)+\Delta$. Note that this index is the last index in the $h$’th batch of $B$.

To see how one can determine this value of $k$ without knowing the weights, first note that the value of $k$ resulting in the answer $r$ to the query $(0,j+|B|,k)$, is precisely the value $k$ such that entry $B[r]$ stores the $k$’th smallest element amongst $B[0],\dots,B[|B|-1],A[0],\dots,A[j]$. We thus need to argue that the number of entries amongst $B[0],\dots,B[|B|-1],A[0],\dots,A[j]$ that stores a value less than or equal to $B[r]=h(\Delta+2)+\Delta+1$ is independent of the weights assigned during updates (ignoring entries of $A$ corresponding to rectangles $R^{(u,v)}_{i}$ containing $(s,t)$ of course).

It is clear that the number of entries amongst $B[0],\dots,B[|B|-1]$ that stores a value less than or equal to $B[r]$ is independent of the weights since it is precisely the $r+1$ entries $B[0],\dots,B[r]$. Thus what remains is to argue that we can also compute the number of such entries amongst $A[0],\dots,A[j]$.

To compute it for $A[0],\dots,A[j]$, examine every rectangle $R^{(u,v)}_{i}$ not containing $(s,t)$. Let $p_{1},p_{2},p_{3},p_{4}$ be the points defined from $R^{(u,v)}_{i}$ above. By the arguments earlier, we have that $(2s,2t)$ dominates either $\{p_{1},p_{2}\},\{p_{1},p_{3}\}$ or $\{p_{1},p_{2},p_{3},p_{4}\}$. Now observe that $\operatorname{rank}_{x}(p_{i})\leq j$ iff $p_{i}.x\leq 2s$ and $\operatorname{rank}_{y}(p_{i})\leq h$ iff $p_{i}.y\leq 2t$. This means that if we perform the update $A[\operatorname{rank}_{x}(p_{i})]\leftarrow\operatorname{rank}_{y}(p_{i})(\Delta+2)$, then $A[\operatorname{rank}_{x}(p_{i})]$ is amongst entries $A[0],\dots,A[j]$ and stores a value less than or equal to $B[r]=h(\Delta+2)+\Delta+1$ iff $(2s,2t)$ dominates $p_{i}$. Also, if we instead performed the update $A[\operatorname{rank}_{x}(p_{i})]\leftarrow|P|(\Delta+2)$, then $A[\operatorname{rank}_{x}(p_{i})]>B[r]$. Now examine the two different update strategies depending on whether the edge $(u,v)$ is assigned the weight $1$ or [math]. The crucial property of our reduction is that precisely one of $\{p_{1},p_{2}\}$ is updated as $A[\operatorname{rank}_{x}(p_{i})]\leftarrow\operatorname{rank}_{y}(p_{i})(\Delta+2)$. Similarly, precisely one of $\{p_{1},p_{3}\}$ is updated as $A[\operatorname{rank}_{x}(p_{i})]\leftarrow\operatorname{rank}_{y}(p_{i})(\Delta+2)$. Finally, precisely two of $\{p_{1},p_{2},p_{3},p_{4}\}$ are updated as $A[\operatorname{rank}_{x}(p_{i})]\leftarrow\operatorname{rank}_{y}(p_{i})(\Delta+2)$. Thus for each $R^{(u,v)}_{i}$ not containing $(s,t)$, we know exactly how many of the corresponding entries of $A$ that is amongst $A[0],\dots,A[j]$ and which store a value less than or equal to $B[r]$, independently of the weights assigned. Thus it follows that if we disregard entries of $A$ corresponding to rectangles containing $(s,t)$, we know the value $k$ such that the selection query $(0,j+|B|,k)$ returns the index $r=h(\Delta+1)+\Delta$.

Let $k$ be the value computed above (without any cell probes), i.e. $k$ is the integer such that the selection query $(0,j+|B|,k)$ returns the index $r=h(\Delta+1)+\Delta$. Recall that $B[r]$ is the last index in the $h$’th chunk of $B$, and this chunk consists of the entries $B[h(\Delta+1)+i]=h(\Delta+2)+i+1$ for $i=0,\dots,\Delta$. To answer the parity searching query $(s,t)$, we run the range selection query $(0,j+|B|,k)$. If the rectangles containing $(s,t)$ were not there, the data structure would return $r$. Now examine the change in the answer to the query $(0,j+|B|,k)$ as we perform the updates corresponding to rectangles $R^{(u,v)}_{i}$ containing $(s,t)$. For such rectangles, $(2s,2t)$ dominates exactly $\{p_{1}\}$ and thus by the arguments above, the four entries $A[\operatorname{rank}_{x}(p_{1})],A[\operatorname{rank}_{x}(p_{2})],A[\operatorname{rank}_{x}(p_{3})],A[\operatorname{rank}_{x}(p_{4})]$ corresponding to $p_{1},p_{2},p_{3}$ and $p_{4}$ have the property that precisely entry $A[\operatorname{rank}_{x}(p_{1})]$ is amongst $A[0],\dots,A[j]$ and stores a value less than or equal to $B[r]$ if the weight assigned to $R^{(u,v)}_{i}$ is $1$. If the weight assigned to $R^{(u,v)}_{i}$ is [math], then none of the entries have that property. Since we are selecting for the $k$’th smallest element in $B[0],\dots,B[|B|-1],A[0],\dots,A[j]$, we get that the first rectangle containing $(s,t)$ and being assigned the weight $1$ changes the answer to the query $(0,j+|B|,k)$ to $r-1$. The next changes the answer to $r-2$ and so forth. Crucially, we know that $(s,t)$ is contained in at most $\Delta$ rectangles. Therefore, the index returned will be exactly $r-m$ where $m$ is the number of rectangles containing $(s,t)$ and being assigned the weight $1$. The chunks of $\Delta+1$ entries thus prevents “overflows” in some sense, ensuring that each rectangle stabbed having weight $1$ decrements the returned index by exactly $1$. Since $r:=h(\Delta+2)+\Delta+1$ is completely determined from the query alone, and the data structure returns the value $r-m$, it follows that we can compute $((r-m)\mod 2)+(r\mod 2)=(-m\mod 2)=(m\mod 2)$, which is the answer to the parity searching query. Note that this computation needs only the parity of the index $(r-m)$ returned by the query. This concludes the proof of Theorem 5.

Acknowledgement

We are very grateful to Rocco Servedio and Oded Regev for insightful discussions on the Peak-to-Average Lemma, and in particular, to Alexander Sherstov for observing and sharing with us the proof of the lower bound (Claim 1 in Appendix B).

Appendix A Proof of Lemma 2 [BCdWZ99]

Lemma 2 (restated).

For any $k$ and $M$ satisfying $2\leq M\leq 2^{O(k)}$, there exists a polynomial $Q=Q_{k,M}(x_{1},\ldots,x_{k})$ such that

1. (i)

$Q$* has total degree $O(\sqrt{k\lg M})$;* 2. (ii)

$|Q(0^{k})|\geq M$; 3. (iii)

$\forall x\in\{0,1\}^{k}\setminus\{0^{k}\},\left|Q(x)\right|\leq 1$; 4. (iv)

The sum of absolute values of all coefficients is at most $\exp(\sqrt{k\lg M})$.

The main idea is to design a lower degree univariate polynomial that takes large value at 0, takes small value at every integer between $1$ and $k$, and has small coefficients. Then apply this polynomial on the Hamming weight of $x$, i.e., $x_{1}+\cdots+x_{k}$. To design such a polynomial, we are going to use Chebyshev polynomials.101010Note that the (standard) Chebyshev polynomials satisfying the second and third properties must have degree $\Theta(\sqrt{k}\lg M)$, which is prohibitively large for our application. The high-level idea is to first take a Chebyshev polynomial $P$ that is large at [math], and is small in $[t,k]$ for some parameter $t$. Then for each integer $i$ between $1$ and $t-1$, we design another polynomial that vanishes at $i$, is not small at [math], and is not large in $[i+1,k]$. Finally, we multiply all these polynomials together, which will produce a polynomial with the claimed properties.

Proof.

If $M\geq 2^{0.05k}$, the statement is trivial, as we can simply set $Q=M\cdot\prod_{i=1}^{k}(1-x_{i})$. Thus, in the following we are going to assume $M\leq 2^{0.05k}$.

In order to bound the sum of coefficients, let us define the following function on polynomials. For a polynomial $P$ on $l$ variables such that $P(x)=\sum_{I\in\mathbb{N}^{l}}\alpha_{I}x^{I}$, define $C(P,s)$ to be the sum of coefficients of $P$ weighted by exponentials of the total degrees:

[TABLE]

Equivalently, $C(P,s)$ is $P$ evaluated on $x_{i}=s$ for all $i$, after replacing each coefficient by its absolute value. Also, the last property in the statement is equivalent to $C(Q,1)\leq\exp(\sqrt{k\lg M})$. It is easy to verify the function $C$ has the following properties:

1. (a)

$C(P_{1}+P_{2},s)\leq C(P_{1},s)+C(P_{2},s)$; 2. (b)

$C(P_{1}\cdot P_{2},s)\leq C(P_{1},s)\cdot C(P_{2},s)$; 3. (c)

For univariate $P_{1}$, $C(P_{1}\circ P_{2},s)\leq C(P_{1},C(P_{2},s))$.

Now consider the (univariate) Chebyshev polynomials $T_{n}$ recursively defined as follows:

• •

$T_{0}(x)=1$;

• •

$T_{1}(x)=x$;

• •

$T_{n+1}(x)=2xT_{n}(x)-T_{n-1}(x)$.

It is well-known that $T_{n}$ is a degree-$n$ polynomial satisfying

• •

$|T_{n}(x)|\leq 1$ for $-1\leq x\leq 1$;

• •

$|T_{n}(-1-\epsilon)|=|T_{n}(1+\epsilon)|\geq\frac{1}{2}e^{n\sqrt{\epsilon}}$ for $0\leq\epsilon<0.1$.

We have $C(T_{0},s)=1$, $C(T_{1},s)=s$, and

[TABLE]

By induction, we have $C(T_{n},s)\leq(2s+1)^{n}$. We are going to construct $Q$ based on Chebyshev polynomials.

Let $t$ be an integer parameter to be set later. Consider the following (univariate) polynomials:

• •

let $P(x)=T_{n}((x-k)/(k-t))$ for $n=\lceil\sqrt{k/t-1}\ln 2M\rceil$;

• •

For each $i=1,2,\ldots,t-1$, let $Q_{i}(x)=(T_{n}((x-k)/(k-i))-T_{n}(-1))/2$ for $n=\lceil 2\sqrt{k/i-1}\rceil$.

It is easy to verify that, as long as $t\leq 0.05k$, we have

• •

$|P(0)|\geq M$, $|P(x)|\leq 1$ for $x\in[t,k]$;

• •

$C(P,k)\leq C(T_{n},2k/(k-t))\leq 6^{\sqrt{k/t-1}\ln 2M+1}$ by Property $(c)$ above;

• •

$|Q_{i}(0)|\geq 1$, $Q_{i}(i)=0$ and $|Q_{i}(x)|\leq 1$ for $x\in[i+1,k]$;

• •

$C(Q_{i},k)\leq(C(T_{n},2k/(k-i))+1)/2\leq 6^{2\sqrt{k/i-1}+1}$.

Finally, we are going to set $t=\lg M(\leq 0.05k)$, and define the polynomial $Q$ as follows:

[TABLE]

In the following, we prove $Q$ has the claimed properties.

1. (i)

The degree of $Q$ is

[TABLE] 2. (ii)

$|Q(0^{k})|=|P(0)\cdot Q_{1}(0)\cdots Q_{t-1}(0)|\geq M$. 3. (iii)

For $x$ such that $x_{1}+\cdots+x_{k}=|x|<t$, we have $Q(x)=0$, since $Q_{|x|}(x)=0$. Otherwise, $|P(x)|\leq 1$ and $|Q_{i}(x)|\leq 1$. We also have $|Q(x)|\leq 1$. 4. (iv)

By Property $(b)$ above, we have $C(P\cdot Q_{1}\cdots Q_{t-1},k)\leq\exp(\sqrt{k\lg M})$. Thus, by Property $(c)$, we have

[TABLE]

as claimed.

∎

Appendix B Tightness of the Peak-to-Average Lemma

In this section we show that Lemma 1 is tight, in the sense that there is a function $f:\{-1,1\}^{k}\mapsto\mathbb{R}$ satisfying the premise of the lemma yet the average value of $f$ conditioned on any subset of $o(\sqrt{k\lg 1/\epsilon})$ coordinates is [math]. In other words, conditioning on $o(\sqrt{k\lg 1/\epsilon})$ coordinates of $f$ provides no advantage at all in predicting the value of $f$, despite the fact that $\|f\|_{\infty}\geq\epsilon$.

The key for constructing our tight counter example is the following claim, which relies on the well known fact that the $\epsilon$-approximate degree of the $\mathsf{AND}_{k}$ function is $\Theta(\sqrt{k\lg 1/\epsilon})$. The following elegant corollary was pointed out to us by Alexander Sherstov (private communication):

Claim 1**.**

For every $\epsilon>2^{-O(k)}$, there is a univariate polynomial $Q:[k]\rightarrow\mathbb{R}$ of degree at most $k-\Omega(\sqrt{k\lg 1/\epsilon})$ satisfying

[TABLE]

Let us first see why this claim implies the existence of our desired function $f$. To this end, assume w.l.o.g that $k$ is even and let $Q$ be the univariate polynomial obtained from Claim 1, and define the multivariate polynomial $f_{Q}:\{-1,1\}^{k}\rightarrow\mathbb{R}$ as

[TABLE]

where $|x|:=k/2+\sum_{i=1}^{k}x_{i}/2$ is the number of $1$ coordinates in $x\in\{-1,1\}^{k}$. By Claim 1, the total degree of $Q$ (now as a multivariate polynomial) is still at most $r:=k-\Omega(\sqrt{k\lg 1/\epsilon})$. Writing $Q(|x|)$ as the sum of its monomials and recalling that $x_{i}^{2}=1$ over $\{-1,1\}$, observe that multiplying each monomial by $\prod_{i=1}^{k}x_{i}$ simply “inverts” each monomial $\prod_{i\in S}x_{i}$ to the monomial $\prod_{i\notin S}x_{i}$. Since each monomial of $Q$ was of degree at most $r=k-\Omega(\sqrt{k\lg 1/\epsilon})$, this implies that all monomials of $f_{Q}$ are of degree at least $k-r=\Omega(\sqrt{k\lg 1/\epsilon})$. In particular, it follows that for any subset of coordinates $Y\subset[k]$, $|Y|=o\left(\sqrt{k\cdot\lg 1/\epsilon}\right)$, the average value of $f_{Q}$ conditioned on coordinates in $Y$ is

[TABLE]

since each monomial of $f_{Q}$ is of degree $\Omega(\sqrt{k\lg 1/\epsilon})$, hence conditioning on any $o\left(\sqrt{k\cdot\lg 1/\epsilon}\right)$ of its coordinates has [math] correlation with the value of the monomial.

To see why $f_{Q}$ satisfies the two premises of Lemma 1, first observe that the conclusion of Claim 1 implies that $\|f_{Q}\|_{1}=\sum_{i=0}^{k}{k\choose i}\cdot|Q(i)|=1$, as required by the first condition of the lemma. To see why the second premise of the lemma holds, we have

[TABLE]

where the last inequality follows from Claim 1. In particular, $\max_{x}|f_{Q}(x)|\geq|f_{Q}(-1,\ldots,-1)|\geq\epsilon$, as the premise of the Peak-to-Average Lemma requires. Hence, to complete the proof of our lower bound, it remains to prove Claim 1.

Proof of Claim 1.

Thought the proof we represent polynomials $P:[k]\mapsto\mathbb{R}$ by their corresponding “truth tables” over the domain $[k]$ (i.e., a vector in $\mathbb{R}^{k+1}$). For example, the function $\mathsf{AND}_{k}:[k]\mapsto\{0,1\}$ is represented by the vector $(1,0,\ldots,0)$. Denote by $\mathcal{B}_{\epsilon}(\mathsf{AND}_{k})$ the $\ell_{\infty}$ ball of radius $\epsilon$ centered at $(1,0,\ldots,0)$, i.e., the convex set of (truth-tables of) functions that point-wise $\epsilon$-approximate the $\mathsf{AND}_{k}$ function on $[k]$, and denote by $\mathcal{P}_{d}$ the set of all (truth-tables of) real polynomials of degree $\leq d$ over the domain $[k]$. It is not hard to verify that under our representation, the set $\mathcal{P}_{d}$ is also convex (since degree-$d$ polynomials are closed under convex combinations). It is well known (e.g., Theorem 3 in [BCdWZ99]) that the lowest degree of a univariate real polynomial that point-wise approximates the $\mathsf{AND}_{k}$ function to within additive error $\epsilon$ is $\Theta(\sqrt{k\lg 1/\epsilon})$ (for every $\epsilon>2^{-O(k)}$), hence if we set $d=\frac{1}{C}\cdot\sqrt{k\lg 1/\epsilon}$ for a large enough constant $C$, the two (convex) sets $\mathcal{B}_{\epsilon}(\mathsf{AND}_{k})$ and $\mathcal{P}_{d}$ are disjoint. In this case, Farkas’ Lemma implies that there is some separating hyperplane $\psi\in\mathbb{R}^{k+1}$ of unit norm ($\|\psi\|_{1}=1$) and some $b\geq 0$, for which: $(i)\;\langle P,\psi\rangle\leq b$ for all $P\in\mathcal{P}_{d}$; and $(ii)\;\langle g,\psi\rangle>b\geq 0$ for all $g\in\mathcal{B}_{\epsilon}(\mathsf{AND}_{k})$. Note that the first condition actually implies that $\psi$ must be orthogonal to all of $\mathcal{P}_{d}$, i.e., $\langle P,\psi\rangle=0$, since degree-$d$ polynomials are closed under scaling and negations (so if $\langle P,\psi\rangle=b\neq 0$ for some degree-$d$ polynomial $P$, then $\langle c\cdot P,\psi\rangle=c\cdot b$ for any $c>0$, and $c\cdot P\in\mathcal{P}_{d}$). Moreover, the second condition in particular holds for the vector $g^{*}\in\mathcal{B}_{\epsilon}(\mathsf{AND}_{k})$ defined by $g^{*}(i):=-sgn(\psi(i))\cdot\epsilon$ for every $i\neq 0$ and $g^{*}(0):=1-\epsilon$, in which case $\langle g^{*},\psi\rangle=(1-\epsilon)\psi(0)-\epsilon\cdot\sum_{i=0}^{k}\psi(i)=|\psi(0)|-\epsilon\cdot\|\psi\|_{1}$. In conclusion, from the above two conditions we can derive that:

1. (1)

$\langle P,\psi\rangle=0$ for all $P\in\mathcal{P}_{d}$; 2. (2)

$|\psi(0)|>\epsilon\cdot\|\psi\|_{1}$ ; 3. (3)

$\|\psi\|_{1}=1$ .

It is a well known fact111111This is a straightforward corollary of the combinatorial identity $\sum_{i}{k\choose i}(-1)^{i}(a_{0}+a_{1}i+a_{2}i^{2}\ldots+a_{k}i^{k})=(-1)^{k}k!a_{k}$ (for arbitrary $(a_{0},\ldots,a_{k})$, see e.g., Equation (5.42) in [GKP94]). Indeed, for any $P\in\mathcal{P}_{d}$ and $\psi$ of the form (11), $\sum_{i}P(i)\cdot\psi(i)={k\choose i}(-1)^{i}(Q\cdot P)(i)$, so whenever $deg(Q)<k-d$, we have $deg(P\cdot Q)<k$ hence the $k$th coefficient is [math] and the above identity implies $\langle P,\psi\rangle=0$ for any $P\in P_{d}$. The other direction follows from the observation that the orthogonal subspace to $\mathcal{P}_{d}$ has dimension $k-d$, and indeed the space spanned by the polynomials in (11) (ranging over all $Q$’s of degree at most $k-d$) has the latter dimension. that every $\psi$ that satisfies (1) must be of the form:

[TABLE]

for all $i\in[k]$, where $Q$ is some degree $\leq(k-d)$-polynomial (again, recall we are representing polynomials by their corresponding truth tables). Substituting this fact in (2), we conclude that there is a polynomial of degree at most $k-d=k-\Omega(\sqrt{k\lg 1/\epsilon})$, satisfying

[TABLE]

since $\sum_{i=0}^{k}\left|{k\choose i}\cdot Q(i)\right|=\|\psi\|_{1}=1$, as claimed. ∎