The Query Complexity of a Permutation-Based Variant of Mastermind

TL;DR

This paper investigates the number of queries needed to identify a secret permutation and binary string in a variant of Mastermind, establishing tight bounds for both deterministic and randomized strategies.

Contribution

It provides the first tight bounds on the query complexity for this permutation-based Mastermind variant, connecting it to black-box optimization problems.

Findings

Deterministic query complexity is Θ(n log n).

Randomized query complexity is Θ(n log log n).

The results match upper and lower bounds, confirming optimal strategies.

Abstract

We study the query complexity of a permutation-based variant of the guessing game Mastermind. In this variant, the secret is a pair $(z,\pi)$ which consists of a binary string $z \in \{0,1\}^n$ and a permutation $\pi$ of $[n]$. The secret must be unveiled by asking queries of the form $x \in \{0,1\}^n$. For each such query, we are returned the score \[ f_{z,\pi}(x):= \max \{ i \in [0..n]\mid \forall j \leq i: z_{\pi(j)} = x_{\pi(j)}\}\,;\] i.e., the score of $x$ is the length of the longest common prefix of $x$ and $z$ with respect to the order imposed by $\pi$. The goal is to minimize the number of queries needed to identify $(z,\pi)$. This problem originates from the study of black-box optimization heuristics, where it is known as the \textsc{LeadingOnes} problem. In this work, we prove matching upper and lower bounds for the deterministic and randomized query complexity of this game, which are $\Theta(n \log n)$ and $\Theta(n \log \log n)$, respectively.

Equations

$$f_{z,\pi }(x):=\max \{i\in [\mathrm{0..}n]\mid \forall j\le i:z_{\pi (j)}=x_{\pi (j)}\};$$

$$f_{z,\pi }(x):=\max \{i\in [\mathrm{0..}n]\mid \forall j\le i:z_{\pi (j)}=x_{\pi (j)}\}.$$

$$\left\{\text{textsc}LeadingOnes\circ \sigma \mid \sigma \text{ is an automorphism of }\{0,1{\}}^n\right\}=\left\{f_{z,\pi }\mid z\in \{0,1{\}}^n,\pi \in S_n\right\},$$

$$\underset{A}{\inf }\underset{f\in \mathcal{F}}{\max }E[T_A(f)],$$

$$f_{z,\pi }:\{0,1{\}}^n\to [\mathrm{0..}n],x\mapsto \max \{i\in [\mathrm{0..}n]\mid \forall j\le i:z_{\pi (j)}=x_{\pi (j)}\}.$$

$$\prod _{1\le i\le n}\left(|V_i|-|\{j<i\mid V_j\subseteq V_i\}|\right).$$

$$\sum _{h=0}^{\log k}ck/2^h<2ck,$$

$$\displaystyle\frac{q}{\alpha_{t}}\bigg{(}\frac{\alpha_{t}\log n}{\log\alpha_{t}}+\frac{\alpha_{t}}{\alpha_{t-1}}\bigg{(}\frac{\alpha_{t-1}dc(\log\alpha_{t}-\log\alpha_{t-1})}{\log\alpha_{t-1}}$$

$$\displaystyle\quad\quad\quad\quad\quad\quad\quad\quad\quad+\frac{\alpha_{t-1}}{\alpha_{t-2}}\bigg{(}\ldots+\frac{\alpha_{2}}{\alpha_{1}}\bigg{(}\frac{\alpha_{1}dc(\log\alpha_{2}-\log\alpha_{1})}{\log\alpha_{1}}+\alpha_{1}d\log\alpha_{1}\bigg{)}\bigg{)}\bigg{)}\bigg{)}$$

$$\le$$

$$\sum _{i=1}^t\frac{n^2}{(n-q)({\alpha }_i^{d-1}-1)}\left(1+\frac{{\alpha }_{i}dc(\log {\alpha }_{i+1}-\log {\alpha }_i)}{\log {\alpha }_i}\right)$$

$$ndc\left(\frac{\log n}{\log {\alpha }_t}+2(t-1)+\log \log n-t\right)=O(n\log \log n)$$

$$\frac{n^{2}dc}{n-q}\sum _{i=1}^t{\alpha }_i^{-(d-3)}=O(n\log n)\sum _{i=0}^{t-1}{\alpha }_1^{-(d-3)2^i}<O(n\log n)({\alpha }_1^{d-3}-1{)}^{-1}=O(n),$$

$$\frac{1}{n-k}\sum _{j=\ell }^t\frac{n{\alpha }_j}{{\alpha }_j^d}=\frac{n}{n-k}\sum _{j=\ell }^t\frac{1}{{\alpha }_j^{d-1}}.$$

$$\frac{n}{n-k}\sum _{j=1}^{t-\ell }{\left(\frac{1}{{\alpha }_{\ell }^{d-1}}\right)}^j<\frac{n}{n-k}{\left({\alpha }_{\ell }^{d-1}-1\right)}^{-1}.$$

$$qn(n-q{)}^{-1}({\alpha }_{\ell }^{d-1}-1{)}^{-1}\le n^2(n-q{)}^{-1}{\left({\alpha }_{\ell }^{d-1}-1\right)}^{-1}.$$

$$V^{w_{j}}_{i}=\left\{\begin{array}[]{l l}V^{v}_{i}\cap P^{v}_{i(\textrm{ mod }2)}&\quad\text{if }i\leq j\,,\\ V^{v}_{i}\cap P^{v}_{j(\textrm{ mod }2)}&\quad\text{if }i=j+1\,,\\ V^{v}_{i}&\quad\text{if }i>j+1.\\ \end{array}\right.$$

$$\phi (v)=\log \log \frac{2n}{|V_{{\max }_v+1}^v|-{\mathrm{Con}}_v}+\sum _{j\in A_v}\log \log \frac{2n}{|V_j^v|},$$

$${\epsilon }_i=|V_i^v(1)|/|V_i^v|,$$

$${\epsilon }_i=|V_i^v(0)|/|V_i^v|.$$

$${\epsilon }_i^{\prime }:=\mathrm{Pr}[a_i\le F(x_v)<a_{i+1}-1|\Pi \in S_v\wedge F(x_v)\ge a_i].$$

$$|V_{a_i}^{w_j}|=(1-{\epsilon }_i)|V_{a_i}^v|$$

$$|V_{a_i}^{w_j}|={\epsilon }_i|V_{a_i}^v|$$

$$|V_{a_i}^{w_j}|=|V_{a_i}^v|$$

$$\mathrm{Pr}[F(x_v)=a_i-1|\Pi \in S_v]\le {\epsilon }_i{\Pi }_{j=1}^{i-1}(1-{\epsilon }_j)(1-{\epsilon }_j^{\prime })$$

$$\mathrm{Pr}[a_i\le F(x_v)<a_{i+1}-1|\Pi \in S_v]\le {\epsilon }_i^{\prime }(1-{\epsilon }_i){\Pi }_{j=1}^{i-1}(1-{\epsilon }_j)(1-{\epsilon }_j^{\prime }).$$

$$\sum _{j\le n/4}\mathrm{Pr}[F(x_v)=j\mid \Pi \in S_v](\phi (w_j)-\phi (v))\le \sum _{j\le n/4}{\epsilon }_j(\phi (w_j)-\phi (v)).$$

$$|V_i^{w_j}|=(1-\Theta (1/n))|V_i^v|$$

$$|V_i^{w_j}|=\Theta (|V_i^v|/n)$$

$$|V_i^{w_j}|=|V_i^v|$$

$$\sum _{j\le n/4}{\epsilon }_j\log \left(\frac{\log (2n/({\epsilon }_j|V_j^v|)}{\log (2n/|V_j^v|)}\right)=\sum _{j\le n/4}{\epsilon }_j\log \left(1+\frac{\log (1/{\epsilon }_j)}{\log (2n/|V_j^v|)}\right)\le \sum _{j\le n/4}{\epsilon }_j\log \frac{\log (1/{\epsilon }_j)}{\log (2n/|V_j^v|)},$$

$$\sum _{j\le n/4}\Theta \left(\frac{1}{n}\right)\frac{\log (n/4)}{\log (2n/|V_j^v|)}\le \Theta \left(\sum _{i=0}^{\log n-1}\frac{\log (n/4)}{2^i\log (n/2^i)}\right).$$

$$E[\phi (w_{{\text{i}}_v})-\phi (v)\mid \Pi \in S_v]=\sum _{a=0}^n\mathrm{Pr}[F(x_v)=a\mid \Pi \in S_v](\phi (w_a)-\phi (v)).$$

Full text

[TABLE]

i.e., the score of $x$ is the length of the longest common prefix of $x$ and $z$ with respect to the order imposed by $\pi$. The goal is to minimize the number of queries needed to identify $(z,\pi)$. This problem originates from the study of black-box optimization heuristics, where it is known as the LeadingOnes problem.

In this work, we prove matching upper and lower bounds for the deterministic and randomized query complexity of this game, which are $\Theta(n\log n)$ and $\Theta(n\log\log n)$, respectively.

1 Introduction

Query complexity, also referred to as decision tree complexity, is one of the most basic models of computation. We aim at learning an unknown object (a secret) by asking queries of a certain type. The cost of the computation is the number of queries made until the secret is unveiled. All other computation is free. Query complexity is one of the standard measures in computational complexity theory.

A related performance measure can be found in the theory of black-box optimization, where we are asked to optimize a function $f:S\rightarrow\mathbb{R}$ without having access to it other than by evaluating (“querying”) the function values $f(x)$ for solution candidates $x\in S$. In black-box optimization, the performance of an algorithm on a class $\mathcal{F}$ of functions is measured by the number of function evaluations that are needed until, for any unknown function $f\in\mathcal{F}$, an optimal search point is evaluated for the first time.

1.1 Problem Description and Summary of Main Results

In this work, we consider the query complexity of the following problem.

Let $S_{n}$ denote the set of permutations of $[n]:=\{1,\ldots,n\}$; let $[0..n]:=\{0,1,\ldots,n\}$. Our problem is that of learning a hidden permutation $\pi\in S_{n}$ together with a hidden bit-string $z\in\{0,1\}^{n}$ through queries of the following type. A query is again a bit-string $x\in\{0,1\}^{n}$. As answer we receive the length of the longest common prefix of $x$ and $z$ in the order of $\pi$, which we denote by

[TABLE]

We call this problem the HiddenPermutation problem. It can be viewed as a guessing game like the well-known Mastermind problem (cf. Section 1.2); however, the secret is now a pair $(z,\pi)$ and not just a string. The problem is a standard benchmark problem in black-box optimization.

It is easy to see (see Section 2) that learning one part of the secret (either $z$ or $\pi$) is no easier (up to $O(n)$ questions) than learning the full secret. It is also not difficult to see that $O(n\log n)$ queries suffice deterministically to unveil the secret (see Section 3). Doerr and Winzen [DW12] showed that randomization allows to beat this bound. They gave a randomized algorithm with $O(n\log n/\log\log n)$ expected complexity. The information-theoretic lower bound is only $\Theta(n)$ as the answer to each query is a number between zero and $n$ and hence may reveal as many as $\log n$ bits. We show that

1. (1)

the deterministic query complexity is $\Theta(n\log n)$, cf. Section 3, and that 2. (2)

the randomized query complexity is $\Theta(n\log\log n)$, cf. Sections 4 and 5.

Both upper bound strategies are efficient, i.e., they can be implemented in polynomial time. The lower bound is established by a (standard) adversary argument in the deterministic case and by a potential function argument in the randomized case. We remark that for many related problems (e.g., sorting, Mastermind, and many coin weighing problems) the asymptotic query complexity, or the best known lower bound for it, equals the information-theoretic lower bound. For our problem, deterministic and randomized query complexity differ, and the randomized query complexity exceeds the information theoretic lower bound. The randomized upper and lower bound require non-trivial arguments. In Section 2 we derive auxiliary results, of which some are interesting in their own right. For example, it can be decided efficiently whether a sequence of queries and answers is consistent.

A summary of the results presented in this work previously appeared in [AAD*+*13].

1.2 Origin of the Problem and Related Work

The HiddenPermutation problem can be seen as a guessing game. An archetypal and well-studied representative of guessing games is Mastermind, a classic two-player board game from the Seventies. In the Mastermind game, one player chooses a secret code $z\in[k]^{n}$. The role of the second player is to identify this code using as few guesses as possible. For each guess $x\in[k]^{n}$ she receives the number of positions in which $z$ and $x$ agree (and in some variants also the number of additional “colors” which appear in both $x$ and $z$, but in different positions).

The Mastermind game has been studied in different context. Among the most prominent works is a study of Erdős and Rényi [ER63], where the 2-color variant of Mastermind is studied in the context of a coin-weighing problem. Their main result showed that the query complexity of this game is $\Theta(n/\log n)$. This bound has subsequently been generalized in various ways, most notably to the Mastermind game with $k=n$ colors. Using similar techniques as Erdős and Rényi, Chvátal [Chv83] showed an upper bound of $O(n\log n)$ for the query complexity of this game. This bound has been improved $O(n\log\log n)$ [DDST16]. The best known lower bound is the information-theoretic linear one. This problem is open for more than 30 years.

Our permutation-based variant of Mastermind has its origins in the field of evolutionary computation. There, the LeadingOnes function $\{0,1\}^{n}\rightarrow[0..n],x\mapsto\max\{i\in[0..n]\mid\forall j\leq i:x_{j}=1\}\,$ counting the number of initial ones in a binary string of length $n$, is a commonly used benchmark function both for experimental and theoretical analyses (e.g., [Rud97]). It is studied as one of the simplest examples of a unimodal non-separable function.

An often desired feature of general-purpose optimization algorithms like genetic and evolutionary algorithms is that they should be oblivious of problem representation. In the context of optimizing functions of the form $f:\{0,1\}^{n}\rightarrow\mathbb{R}$, this is often formalized as an unbiasedness restriction, which requires that the performance of an unbiased algorithm is identical for all composed functions $f\circ\sigma$ of $f$ with an automorphism $\sigma$ of the $n$-dimensional hypercube. Most standard evolutionary algorithms as well many other commonly used black-box heuristics like local search variants, simulated annealing, etc. are unbiased. Most of them query an average number of $\Theta(n^{2})$ solution candidates until they find the maximum of LeadingOnes, see, e.g., [DJW02].

It is not difficult to see that

[TABLE]

showing that HiddenPermutation generalizes the LeadingOnes function by indexing the bits not from left to right, but by an arbitrary permutation, and by swapping the interpretation of [math] and $1$ for indices $i$ with $z_{i}=0$.

The first to study the query complexity of HiddenPermutation were Droste, Jansen, and Wegener [DJW06], who introduced the notion of black-box complexity as a measure for the difficulty of black-box optimization problems. As mentioned, the main objective in black-box optimization is the identification of an optimal solution using as few function evaluations as possible. Denoting by $T_{A}(f)$ the number of queries needed by algorithm $A$ until it queries for the first time a search point $x\in\arg\max f$ (this is the so-called runtime or optimization time of $A$), the black-box complexity of a collection $\mathcal{F}$ of functions is

[TABLE]

the best (among all algorithms) worst-case (with respect to all $f\in\mathcal{F}$) expected runtime. Black-box complexity is essentially query-complexity, with a focus on optimization rather than on learning. A survey on this topic can be found in [Doe18].

Droste et al. [DJW06] considered only the [math]/$1$-invariant class $\left\{f_{z,\operatorname{id}}\mid z\in\{0,1\}^{n}\right\}$ (where $\operatorname{id}$ denotes the identity permutation). They showed that the black-box complexity of this function class is $n/2\pm o(n)$.

The first bound for the black-box complexity of the full class HiddenPermutation was presented in [LW12] by Lehre and Witt, who showed that any so-called unary unbiased black-box algorithm needs $\Omega(n^{2})$ steps, on average, to solve the HiddenPermutation problem. In [DJK*+*11] it was then shown that already with binary distributions one can imitate a binary search algorithm (similar to the one presented in the proof of Theorem 3.1), thus yielding an $O(n\log n)$ black-box algorithm for the HiddenPermutation problem. The algorithm achieving the $O(n\log n/\log\log n)$ bound mentioned in Section 1.1 can be implemented as a ternary unbiased one [DW12]. This bound, as mentioned above, was the previously best known upper bound for the black-box complexity of HiddenPermutation.

In terms of lower bounds, the best known bound was the linear information-theoretic one. However, more recently a stronger lower bound has been proven to hold for a restricted class of algorithms. More precisely, Doerr and Lengler [DL18] recently proved that the so-called $(1+1)$ elitist black-box complexity is $\Omega(n^{2})$, showing that any algorithm beating this bound has to keep in its memory more than one previously evaluated search point or has to make use of information hidden in non-optimal search points. It is conjectured in [DL18] that the (1+1) memory-restriction alone (which allows an algorithm to store only one previously queried search point in its memory and to evaluate in each iteration only one new solution candidate) already causes the $\Omega(n^{2})$ bound, but this conjecture stands open. We note that many local search variants, including simulated annealing, are $(1+1)$ memory-restricted.

2 Preliminaries

For all positive integers $k\in\mathbb{N}$ we define $[k]:=\{1,\ldots,k\}$ and $[0..k]:=[k]\cup\{0\}$. By $e^{n}_{k}$ we denote the $k$th unit vector $(0,\ldots,0,1,0,\ldots,0)$ of length $n$. For a set $I\subseteq[n]$ we define $e^{n}_{I}:=\sum_{i\in I}{e^{n}_{i}}=\oplus_{i\in I}{e^{n}_{i}}$, where $\oplus$ denotes the bitwise exclusive-or. We say for two bitstrings $x,y\in\{0,1\}^{n}$ that we created $y$ from $x$ by flipping $I$ or that we created $y$ from $x$ by flipping the entries in position(s) $I$ if $y=x\oplus e^{n}_{I}$. By $S_{n}$ we denote the set of all permutations of $[n]$. For $r\in\mathbb{R}_{\geq 0}$, let $\lceil r\rceil:=\min\{n\in\mathbb{N}_{0}\mid n\geq r\}$. and $\lfloor r\rfloor:=\max\{n\in\mathbb{N}_{0}\mid n\leq r\}$. To increase readability, we sometimes omit the $\lceil\cdot\rceil$ signs; that is, whenever we write $r$ where an integer is required, we implicitly mean $\lceil r\rceil$.

Let $n\in\mathbb{N}$. For $z\in\{0,1\}^{n}$ and $\pi\in S_{n}$ we define

[TABLE]

$z$ and $\pi$ are called the target string and target permutation of $f_{z,\pi}$, respectively. We want to identify target string and permutation by asking queries $x^{i}$, $i=1$, $2$, …, and evaluating the answers (“scores”) $s^{i}=f_{z,\pi}(x^{i})$. We may stop after $t$ queries if there is only a single pair $(z,\pi)\in\{0,1\}^{n}\times S_{n}$ with $s^{i}=f_{z,\pi}(x^{i})$ for $1\leq i\leq t$.

A deterministic strategy for the HiddenPermutation problem is a tree of outdegree $n+1$ in which a query in $\{0,1\}^{n}$ is associated with every node of the tree. The search starts as the root. If the search reaches a node, the query associated with the node is asked, and the search proceeds to the child selected by the score. The complexity of a strategy on input $(z,\pi)$ is the number of queries required to identify the secret, and the complexity of a deterministic strategy is the worst-case complexity of any input. That is, the complexity is the height of the search tree.

A randomized strategy is a probability distribution over deterministic strategies. The complexity of a randomized strategy on input $(z,\pi)$ is the expected number of queries required to identify the secret, and the complexity of a randomized strategy is the worst-case complexity of any input. The probability distribution used for our randomized upper bound is a product distribution in the following sense: a probability distribution over $\{0,1\}^{n}$ is associated with every node of the tree. The search starts as the root. In any node, the query is selected according to the probability distribution associated with the node, and the search proceeds to the child selected by the score.

We remark that knowing $z$ allows us to determine $\pi$ with $n-1$ queries $z\oplus e^{n}_{i}$, $1\leq i<n$. Observe that $\pi^{-1}(i)$ equals $f_{z,\pi}(z\oplus e^{n}_{i})+1$. Conversely, knowing the target permutation $\pi$ we can identify $z$ in a linear number of guesses. If our query $x$ has a score of $k$, all we need to do next is to query the string $x^{\prime}$ that is created from $x$ by flipping the entry in position $\pi(k+1)$. Thus, learning one part of the secret is no easier (up to $O(n)$ questions) than learning the full.

A simple information-theoretic argument gives an $\Omega(n)$ lower bound for the deterministic query complexity and, together with Yao’s minimax principle [Yao77], also for the randomized complexity. The search space has size $2^{n}n!$, since the unknown secret is an element of $\{0,1\}^{n}\times S_{n}$. A deterministic strategy is a tree with outdegree $n+1$ and $2^{n}n!$ leaves. The maximal and average depth of any such tree is $\Omega(n)$.

Let $\mathcal{H}:=(x^{i},s^{i})_{i=1}^{t}$ be a sequence of queries $x^{i}\in\{0,1\}^{n}$ and scores $s^{i}\in[0..n]$. We call $\mathcal{H}$ a guessing history. A secret $(z,\pi)$ is consistent with $\mathcal{H}$ if $f_{z,\pi}(x^{i})=s^{i}$ for all $i\in[t]$. $\mathcal{H}$ is feasible if there exists a secret consistent with it.

An observation crucial in our proofs is the fact that a vector $(V_{1},\ldots,V_{n})$ of subsets of $[n]$, together with a top score query $(x^{*},s^{*})$, captures the total knowledge provided by a guessing history $\mathcal{H}=(x^{i},s^{i})_{i=1}^{t}$ about the set of secrets consistent with $\mathcal{H}$. We will call $V_{j}$ the candidate set for position $j$; $V_{j}$ will contain all indices $i\in[n]$ for which the following simple rules (1) to (3) do not rule out that $\pi(j)$ equals $i$. Put differently, $V_{j}$ contains the set of values that are still possible images for $\pi(j)$, given the previous queries.

Theorem 2.1**.**

Let $t\in\mathbb{N}$, and let $\mathcal{H}=(x^{i},s^{i})_{i=1}^{t}$ be a guessing history. Construct the candidate sets $V_{1},\ldots,V_{n}\subseteq[n]$ according to the following rules:

1. (1)

If there are $h$ and $\ell$ with $j\leq s^{h}\leq s^{\ell}$ and $x_{i}^{h}\not=x_{i}^{\ell}$, then $i\not\in V_{j}$. 2. (2)

If there are $h$ and ${\ell}$ with $s=s^{h}=s^{\ell}$ and $x_{i}^{h}\not=x_{i}^{\ell}$, then $i\not\in V_{s+1}$. 3. (3)

If there are $h$ and ${\ell}$ with $s^{h}<s^{\ell}$ and $x_{i}^{h}=x_{i}^{\ell}$, then $i\not\in V_{s^{h}+1}$. 4. (4)

If $i$ is not excluded by one of the rules above, then $i\in V_{j}$.

Furthermore, let $s^{*}:=\max\{s^{1},\ldots,s^{t}\}$ and let $x^{*}=x^{j}$ for some $j$ with $s^{j}=s^{*}$.

Then a pair $(z,\pi)$ is consistent with $\mathcal{H}$ if and only if

1. (a)

$f_{z,\pi}(x^{*})=s^{*}$* and* 2. (b)

$\pi(j)\in V_{j}$* for all $j\in[n]$.*

Proof.

Let $(z,\pi)$ satisfy conditions (a) and (b). We show that $(z,\pi)$ is consistent with $\mathcal{H}$. To this end, let $h\in[t]$, let $x=x^{h}$, $s=s^{h}$, and $f:=f_{z,\pi}(x)$. We need to show $f=s$.

Assume $f<s$. Then $z_{\pi(f+1)}\neq x_{\pi(f+1)}$. Since $f+1\leq s^{*}$, this together with (a) implies $x_{\pi(f+1)}\neq x^{*}_{\pi(f+1)}$. Rule (1) yields $\pi(f+1)\notin V_{f+1}$; a contradiction to (b).

Similarly, if we assume $f>s$, then $x_{\pi(s+1)}=z_{\pi(s+1)}$. We distinguish two cases. If $s<s^{*}$, then by condition (a) we have $x_{\pi(s+1)}=x^{*}_{\pi(s+1)}$. By rule (3) this implies $\pi(s+1)\notin V_{s+1}$; a contradiction to (b).

On the other hand, if $s=s^{*}$, then $x_{\pi(s+1)}=z_{\pi(s+1)}\neq x^{*}_{\pi(s+1)}$ by (a). Rule (2) implies $\pi(s+1)\notin V_{\pi(s+1)}$, again contradicting (b).

Necessity is trivial. ∎

We may also construct the sets $V_{j}$ incrementally. The following update rules are direct consequences of Theorem 2.1. In the beginning, let $V_{j}:=[n]$, $1\leq j\leq n$. After the first query, record the first query as $x^{*}$ and its score as $s^{*}$. For all subsequent queries, do the following: Let $I$ be the set of indices in which the current query $x$ and the current best query $x^{*}$ agree. Let $s$ be the score of $x$ and let $s^{*}$ be the score of $x^{*}$.

1. Rule 1:

If $s<s^{*}$, then $V_{i}\leftarrow V_{i}\cap I$ for $1\leq i\leq s$ and $V_{s+1}\leftarrow V_{s+1}\setminus I$. 2. Rule 2:

If $s=s^{*}$, then $V_{i}\leftarrow V_{i}\cap I$ for $1\leq i\leq s^{*}+1$. 3. Rule 3:

If $s>s^{*}$, then $V_{i}\leftarrow V_{i}\cap I$ for $1\leq i\leq s^{*}$ and $V_{s^{*}+1}\leftarrow V_{s^{*}+1}\setminus I$. We further replace $s^{*}\leftarrow s$ and $x^{*}\leftarrow x$.

It is immediate from the update rules that the sets $V_{j}$ form a laminar family; i.e., for $i<j$ either $V_{i}\cap V_{j}=\emptyset$ or $V_{i}\subseteq V_{j}$. As a consequence of Theorem 2.1 we obtain a polynomial time test for the feasibility of histories. It gives additional insight in the meaning of the candidate sets $V_{1},\ldots,V_{n}$.

Theorem 2.2**.**

It is decidable in polynomial time whether a guessing history is feasible. Furthermore, we can efficiently compute the number of pairs $(z,\pi)\leavevmode\nobreak\ \in\leavevmode\nobreak\ \{0,1\}^{n}\leavevmode\nobreak\ \times\leavevmode\nobreak\ S_{n}$ consistent with it.

Proof.

We first show that feasibility can be checked in polynomial time. Let $\mathcal{H}=(x^{i},s^{i})_{i=1}^{t}$ be given. Construct the sets $V_{1},\ldots,V_{n}$ as described in Theorem 2.1. Now construct a bipartite graph $G(V_{1},\ldots,V_{n})$ with node set $[n]$ on both sides. Connect $j$ to all nodes in $V_{j}$ on the other side. Permutations $\pi$ with $\pi(j)\in V_{j}$ for all $j$ are in one-to-one correspondence to perfect matchings in this graph. We recall that perfect matchings can be computed efficiently in bipartite graphs, e.g., using the Ford-Fulkerson algorithm, which essentially treats the matching problem as a network flow problem. If there is no perfect matching, the history in infeasible. Otherwise, let $\pi$ be any permutation with $\pi(j)\in V_{j}$ for all $j$. We next construct $z$. We use the obvious rules:

1. (a)

If $i=\pi(j)$ and $j\leq s^{h}$ for some $h\in[t]$ then set $z_{i}:=x_{i}^{h}$. 2. (b)

If $i=\pi(j)$ and $j=s^{h}+1$ for some $h\in[t]$ then set $z_{i}:=1-x_{i}^{h}$. 3. (c)

If $z_{i}$ is not defined by one of the rules above, set it to an arbitrary value.

We need to show that these rules do not lead to a contradiction. Assume otherwise. There are three ways, in which we could get into a contradiction. There is some $i\in[n]$ and some $x^{h},x^{\ell}\in\{0,1\}^{n}$

1. (1)

setting $z_{i}$ to opposite values by rule (a) 2. (2)

setting $z_{i}$ to opposite values by rule (b) 3. (3)

setting $z_{i}$ to opposite values by rules (b) applied to $x^{h}$ and rule (a) applied to $x^{\ell}$.

In each case, we readily derive a contradiction. In the first case, we have $j\leq s^{h}$, $j\leq s^{\ell}$ and $x_{i}^{h}\not=x_{i}^{\ell}$. Thus $\pi(j)=i\not\in V_{j}$ by rule (1). In the second case, we have $j=s^{h}+1=s^{\ell}+1$ and $x_{i}^{h}\not=x^{\ell}_{i}$. Thus $i\not\in V_{j}$ by (2). In the third case, we have $j=s^{h}+1$, $j\leq s^{\ell}$, and $x_{i}^{h}=x_{i}^{\ell}$. Thus $i\not\in V_{j}$ by (3).

Finally, the pair $(z,\pi)$ defined in this way is clearly consistent with the history.

Next we show how to efficiently compute the number of consistent pairs. We recall Hall’s condition for the existence of a perfect matching in a bipartite graph. A perfect matching exists if and only if $|\bigcup_{j\in J}V_{j}|\geq|J|$ for every $J\subseteq[n]$. According to Theorem 2.1, the guessing history $\mathcal{H}$ can be equivalently described by a state $(V_{1},\ldots,V_{s^{*}+1},x^{*},s^{*})$. How many pairs $(z,\pi)$ are compatible with this state?

Once we have chosen $\pi$, there are exactly $2^{n-(s^{*}+1)}$ different choices for $z$ if $s^{*}<n$ and exactly one choice if $s^{*}=n$. The permutations can be chosen in a greedy fashion. We fix $\pi(1),\ldots,\pi(n)$ in this order. The number of choices for $\pi(i)$ equals $|V_{i}|$ minus the number of $\pi(j)$, $j<i$, lying in $V_{i}$. If $V_{j}$ is disjoint from $V_{i}$, $\pi(j)$ never lies in $V_{i}$ and if $V_{j}$ is contained in $V_{i}$, $\pi(j)$ is always contained in $V_{i}$. Thus the number of permutations is equal to

[TABLE]

It is easy to see that the greedy strategy does not violate Hall’s condition. ∎

We note that it is important in the proof of Theorem 2.2 that the sets $V_{j}$ form a laminar family; counting the number of perfect matchings in a general bipartite graph is #P-complete [Val79].

From the proof can also derive which values in $V_{i}$ are actually possible as value for $\pi(i)$. This will be described in the next paragraph, which a reader only interested in the main results can skip without loss. A value $\ell\in V_{i}$ is feasible if there is a perfect matching in the graph $G(V_{1},\ldots,V_{n})$ containing the edge $(i,\ell)$. The existence of such a matching can be decided in polynomial time; we only need to test for a perfect matching in the graph $G\setminus\{i,\ell\}$. Hall’s condition says that there is no such perfect matching if there is a set $J\subseteq[n]\setminus\{i\}$ such that $|\bigcup_{j\in J}V_{j}\setminus\{\ell\}|<|J|$. Since $G$ contains a perfect matching (assuming a consistent history), this implies $|\bigcup_{j\in J}V_{j}|=|J|$; i.e., $J$ is tight for Hall’s condition. We have now shown: Let $\ell\in V_{i}$. Then $\ell$ is infeasible for $\pi(i)$ if and only if there is a tight set $J$ with $i\not\in J$ and $\ell\in\bigcup_{j\in J}V_{j}$. Since the $V_{i}$ form a laminar family, minimal tight sets have a special form; they consist of an $i$ and all $j$ such that $V_{j}$ is contained in $V_{i}$. In the counting formula for the number of permutations such $i$ are characterized by $|V_{i}|-|\{j<i\,\mid\,V_{j}\subseteq V_{i}\}|=1$. In this situation, the elements of $V_{i}$ are infeasible for all $\pi(j)$ with $j>i$. We may subtract $V_{i}$ from each $V_{j}$ with $j>i$.

If Hall’s condition is tight for some $J$, i.e., $|\bigcup_{j\in J}V_{j}|=|J|$, we can easily learn how $\pi$ operates on $J$. We have $V_{j}=[n]$ for $j>s^{*}+1$ and hence the largest index in $J$ is at most $s^{*}+1$. Perturb $x^{*}$ by flipping each bit in $\bigcup_{j\in J}V_{j}$ exactly once. The scores determine the permutation.

3 Deterministic Complexity

We settle the deterministic query complexity of the HiddenPermutation problem. The upper and lower bound match up to a small constant factor. Specifically, we prove

Theorem 3.1**.**

The deterministic query complexity of the HiddenPermutation problem with $n$ positions is $\Theta(n\log n)$.

Proof.

The upper bound is achieved by an algorithm that resembles binary search and iteratively identifies $\pi(1),\ldots,\pi(n)$ and the corresponding bit values $z_{\pi(1)},\ldots,z_{\pi(n)}$: We start by setting the set $V$ of candidates for $\pi(1)$ to $[n]$ and by determinining a string $x$ with score 0; either the all-zero-string or the all-one-string will work. We iteratively reduce the size of $V$ keeping the invariant that $\pi(1)\in V$. We select an arbitrary subset $F$ of $V$ of size $|V|/2$ and create $y$ from $x$ by flipping the bits in $F$. If $f_{z,\pi}(y)=0$, $\pi(1)\not\in F$, if If $f_{z,\pi}(y)>0$, $\pi(1)\in F$. In either case, we essentially halve the size of the candidate set. Continuing in this way, we determine $\pi(1)$ in $O(\log n)$ queries. Once $\pi(1)$ and $z_{\pi(1)}$ are known, we iterate this strategy on the remaining bit positions to determine $\pi(2)$ and $z_{\pi(2)}$, and so on, yielding an $O(n\log n)$ query strategy for identifying the secret. The details are given in Algorithm 3.2.

The lower bound is proved by examining the decision tree of the deterministic query scheme and exhibiting an input for which the number of queries asked is high. More precisely, we show that for every deterministic strategy, there exists an input $(z,\pi)$ such that after $\Omega(n\log n)$ queries the maximal score ever returned is at most $n/2$. This is done by a simple adversarial argument: First consider the root node $r$ of the decision tree. Let $x^{1}$ be the first query. We proceed to the child corresponding to score 1. According to the rules from the preceding section $V_{1}$ to $V_{n}$ are initialized to $[n]$. Let $x$ be the next query asked by the algorithm and let $I$ be the set of indices in which $x$ and $x^{1}$ agree.

1. (1)

If we would proceed to the child corresponding to score [math], then $V_{1}$ would become $V_{1}\setminus I$ and $V_{2}$ would not not change according to Rule 1. 2. (2)

If we would proceed to the child corresponding to score $1$, then $V_{1}$ would become $V_{1}\cap I$ and $V_{2}$ would become $V_{2}\cap I$ according to Rule 2.

We proceed to the child where the size of $V_{1}$ at most halves. Observe that $V_{1}\subseteq V_{2}$ always, and the maximum score is $1$. Moreover, $V_{3}$ to $V_{n}$ are not affected.

We continue in this way until $|V_{1}|=2$. Let $v$ be the vertex of the decision tree reached. Query $x^{*}=x^{1}$ is still the query with maximum score. We choose $i_{1}\in V_{1}$ and $i_{2}\in V_{2}$ arbitrarily and consider the subset of all inputs for which $i_{1}=\pi(1)$, $i_{2}=\pi(2)$, $z_{i_{1}}=x^{*}_{i_{1}}$, and $z_{i_{2}}=1-x^{*}_{i_{2}}$. For all such inputs, the query path followed in the decision tree descends from the root to the node $v$. For this collection of inputs, observe that there is one input for every assignment of values to $\pi(3),\dots,\pi(n)$ different from $i_{1}$ and $i_{2}$, and for every assignment of $0/1$ values to $z_{\pi(3)},\dots,z_{\pi(n)}$. Hence we can recurse on this subset of inputs starting at $v$ ignoring $V_{1},V_{2},\pi(1),\pi(2),z_{\pi(1)}$, and $z_{\pi(2)}$. The setup is identical to what we started out with at the root, with the problem size decreased by 2. We proceed this way, forcing $\Omega(\log n)$ queries for every two positions revealed, until we have returned a score of $n/2$ for the first time. At this point, we have forced at least $n/4\cdot\Omega(\log n)=\Omega(n\log n)$ queries. ∎

4 The Randomized Strategy

We now show that the randomized query complexity is only $O(n\log\log n)$. The randomized strategy overcomes the sequential learning process of the binary search strategy (typically revealing a constant amount of information per query) and instead has a typical information gain of $\Theta(\log n/\log\log n)$ bits per query. In the language of the candidate sets $V_{i}$, we manage to reduce the sizes of many of these sets in parallel, that is, we gain information on several values $\pi(i)$ despite the seemingly sequential way the function $f_{z,\pi}$ offers information. The key to this is using partial information given by the $V_{i}$ (that is, information that does not determine $\pi_{i}$, but only restricts it) to guess with good probability an $x$ with $f_{z,\pi}(x)>s^{*}$.

Theorem 4.1**.**

The randomized query complexity of the HiddenPermutation problem with $n$ positions is $O(n\log\log n)$.

The strategy has two parts. In the first part, we identify the positions $\pi(1),\ldots,\pi(q)$ and the corresponding bit values $z_{\pi(1)},\ldots,z_{\pi(q)}$ for some $q\in n-\Theta(n/\log n)$ with $O(n\log\log n)$ queries. In the second part, we find the remaining $n-q\in\Theta(n/\log n)$ positions and entries using the binary search algorithm with $O(\log n)$ queries per position.

4.1 A High Level View of the First Part

We give a high level view of the first part of our randomized strategy. Here and in the following we denote by $s^{*}$ the current best score, and by $x^{*}$ we denote a corresponding query; i.e., $f_{z,\pi}(x^{*})=s^{*}$. For brevity, we write $f$ for $f_{z,\pi}$.

The goal of any strategy must be to increase $s^{*}$ and to gain more information about $\pi$ by reducing the sets $V_{1},\ldots,V_{s^{*}+1}$. Our strategy carefully balances the two subgoals. If $|\bigcup_{i\leq s^{*}+1}V_{i}|$ is “large”, it concentrates on reducing sets, if $|\bigcup_{i\leq s^{*}+1}V_{i}|$ is “small”, it concentrates on increasing $s^{*}$. The latter will simulatenously reduce $V_{s^{*}+1}$.

We arrange the candidate sets $V_{1}$ to $V_{n}$ into $t+2$ levels 0 to $t+1$, where $t=\Theta(\log\log n)$. Initially, all candidate sets are on level [math], and we have $V_{i}=[n]$ for all $i\in[n]$. The sets in level $i$ have larger index than the sets in level $i+1$. Level $t+1$ contains an initial segment of candidate sets, and all candidate sets on level $t+1$ are singletons, i.e., we have identified the corresponding $\pi$-value. On level $i$, $1\leq i\leq t$, we can have up to $\alpha_{i}$ sets. We also say that the capacity of level $i$ is $\alpha_{i}$. The size of any set on level $i$ is at most $n/\alpha_{i}^{d}$, where $d$ is any constant greater than or equal to $4$. We choose $\alpha_{1}=\log n$, $\alpha_{i}=\alpha_{i-1}^{2}$ for $1\leq i\leq t$ and $t$ maximal such that $\alpha_{t}^{d}\leq n$. Depending on the status (i.e., the fill rate) of these levels, either we try to increase $s^{*}$, or we aim at reducing the sizes of the candidate sets.

The algorithm maintains a counter $s\leq s^{*}$ and strings $x,y\in\{0,1\}^{n}$ with $f(x)=s<f(y)$. The following invariants hold for the candidate sets $V_{1}$ to $V_{n}$:

1. (1)

$\pi(j)\in V_{j}$ for all $j$. 2. (2)

The sets $V_{j}$, $j\leq s$, are pairwise disjoint. 3. (3)

$V_{j}=[n]$ for $j>s$. 4. (4)

$V_{j}\setminus\{\pi(j)\}$ is random. More precisely, there is a set $V_{j}^{*}\subseteq[n]$ such that $\pi(j)\in V_{j}$ and $V_{j}\setminus\{\pi(j)\}$ is a uniformly random subset of $V_{j}^{*}\setminus\{\pi(j)\}$ of size $|V_{j}|-1$.

Our first goal is to increase $s^{*}$ to $\log n$ and to move the sets $V_{1},\ldots,V_{\log n}$ to the first level, i.e., to decrease their size to $n/\alpha_{1}^{d}=n/\log^{d}n$. This is done sequentially. We start by querying $f(x)$ and $f(y)$, where $x$ is arbitrary and $y=x\oplus 1^{n}$ is the bitwise complement of $x$. By swapping $x$ and $y$ if needed, we may assume $f(x)=0<f(y)$. We now run a randomized binary search for finding $\pi(1)$. We choose uniformly at random a subset $F_{1}\subseteq V_{1}$ ($V_{1}=[n]$ in the beginning) of size $|F_{1}|=|V_{1}|/2$. We query $f(y^{\prime})$ where $y^{\prime}$ is obtained from $x$ by flipping the bits in $F_{1}$. If $f(y^{\prime})=0$, we set $V_{1}\leftarrow V_{1}\setminus F_{1}$; we set $V_{1}\leftarrow F_{1}$ otherwise. This ensures $\pi(1)\in V_{1}$ and invariant (4). We stop this binary search once $\pi(2)\not\in V_{1}$ is sufficiently likely; the analysis will show that $\Pr[\pi(2)\in V_{1}]\leq 1/\log^{d}n$ (and hence $|V_{1}|\leq n/\log^{d}n$) for some large enough constant $d$ is a good choice.

We next try to increase $s$ to a value larger than one and to simultaneously decrease the size of $V_{2}$. Let $\{x,y\}=\{y,y\oplus 1_{[n]\setminus V_{1}}\}$. If $\pi(2)\not\in V_{1}$, one of $f(x)$ and $f(y)$ is one and the other is larger than one. Swapping $x$ and $y$ if necessary, we may assume $f(x)=1<f(y)$. We use randomized binary search to reduce the size of $V_{2}$ to $n/\log^{d}n$. The randomized binary search is similar to before. Initially, $V_{2}$ is equal to $V_{2}^{*}=[n]-V_{1}$. At each step we chose a subset $F_{2}\subseteq V_{2}$ of size $|V_{2}|/2$ and we create $y^{\prime}$ from $x$ by flipping the bits in positions $F_{2}$. If $f(y^{\prime})>1$ we update $V_{2}$ to $F_{2}$ and we update $V_{2}$ to $V_{2}\setminus F_{2}$ otherwise. We stop once $|V_{2}|\leq n/\log^{d}n$.

At this point we have $|V_{1}|,|V_{2}|\leq n/\log^{d}n$ and $V_{1}\cap V_{2}=\emptyset$. We hope that $\pi(3)\notin V_{1}\cup V_{2}$, in which case we can increase $s$ to three and move set $V_{3}$ from level [math] to level $1$ by random binary search (the case $\pi(3)\in V_{1}\cup V_{2}$ is called a failure and will be treated separately at the end of this overview).

At some point the probability that $\pi(i)\notin V_{1}\cup\ldots\cup V_{i-1}$ drops below a certain threshold and we cannot ensure to make progress anymore by simply querying $y\oplus([n]\backslash(V_{1}\cup\ldots\cup V_{i-1}))$. This situation is reached when $i=\log n$ and hence we abandon the previously described strategy once $s=\log n$. At this point, we move our focus from increasing $s$ to reducing the size of the candidate sets $V_{1},\ldots,V_{s}$, thus adding them to the second level. More precisely, we reduce their sizes to at most $n/\log^{2d}n=n/\alpha_{2}^{d}$. This reduction is carried out by SizeReduction, which we describe in Section 4.3. It reduces the sizes of the up to $\alpha_{\ell-1}$ candidate sets from some value $\leq n/\alpha_{\ell-1}^{d}$ to the target size $n/{\alpha_{\ell}^{d}}$ of level $\ell$ with an expected number of $O(1)\alpha_{\ell-1}d(\log(\alpha_{\ell})-\log(\alpha_{\ell-1}))/\log(\alpha_{\ell-1})$ queries.

Once the sizes $|V_{1}|,\ldots,|V_{s}|$ have been reduced to at most $n/\log^{2d}n$, we move our focus back to increasing $s$. The probability that $\pi(s+1)\in V_{1}\cup\ldots\cup V_{s}$ will now be small enough (details below), and we proceed as before by flipping in $x$ the entries in the positions $[n]\setminus(V_{1}\cup\ldots\cup V_{s})$ and reducing the size of $V_{s+1}$ to $n/\log^{d}n$. Again we iterate this process until the first level is filled; i.e., until we have $s=2\log n$. As we did with $V_{1},\ldots,V_{\log n}$, we reduce the sizes of $V_{\log n+1},\ldots,V_{2\log n}$ to $n/\log^{2d}n=n/\alpha_{2}^{d}$, thus adding them to the second level. We iterate this process of moving $\log n$ sets from level [math] to level $1$ and then moving them to the second level until $\log^{2}n=\alpha_{2}$ sets have been added to the second level. At this point the second level has reached its capacity and we proceed by reducing the sizes of $V_{1},\ldots,V_{\log^{2}n}$ to at most $n/\log^{4d}n=n/\alpha_{3}^{d}$, thus adding them to the third level.

In total we have $t=O(\log\log n)$ levels. For $1\leq i\leq t$, the $i$th level has a capacity of $\alpha_{i}:=\log^{2^{i-1}}n$ sets, each of which is required to be of size at most $n/\alpha_{i}^{d}$. Once level $i$ has reached its capacity, we reduce the size of the sets on the $i$th level to at most $n/\alpha_{i+1}^{d}$, thus moving them from level $i$ to level $i+1$. When $\alpha_{t}$ sets $V_{i+1},\ldots,V_{i+\alpha_{t}}$ have been added to the last level, level $t$, we finally reduce their sizes to one. This corresponds to determining $\pi(i+j)$ for each $j\in[\alpha_{t}]$.

Failures:

We say that a failure happens if we want to move some set $V_{s+1}$ from level 0 to level 1, but $\pi(s+1)\in V_{1}\cup\ldots\cup V_{s}$. In case of a failure, we immediately stop our attempt of increasing $s$. Rather, we abort the first level and move all sets on the first level to the second one. As before, this is done by calls to SizeReduction which reduce the size of the sets from at most $n/\log^{d}n$ to at most $n/\log^{2d}n$. We test whether we now have $\pi(s+1)\not\in V_{1}\cup\ldots\cup V_{s}$. Should we still have $\pi(s+1)\in V_{1}\cup\ldots\cup V_{s}$, we continue by moving all level $2$ sets to level $3$, and so on, until we finally have $\pi(s+1)\not\in V_{1}\cup\ldots\cup V_{s}$. At this point, we proceed again by moving sets from level [math] to level $1$, starting of course with set $V_{s+1}$. The condition $\pi(s+1)\not\in V_{1}\cup\ldots\cup V_{s}$ will certainly be fulfilled once we have moved $V_{1}$ to $V_{s}$ to level $t+1$, i.e., have reduced them to singletons.

Part 2:

In the second part of Algorithm 4.2 we determine the last $\Theta(n/\log n)$ entries of $z$ and $\pi$. This can be done as follows. When we leave the first phase of Algorithm 4.2, we have $|V_{1}|=\ldots=|V_{q}|=1$ and $f(x)\geq q$. We can now proceed as in deterministic algorithm (Algorithm 3.2) and identify each of the remaining entries with $O(\log n)$ queries. Thus the total number of queries in Part 2 is linear.

Our strategy is formalized by Algorithm 4.2. In what follows, we first present the two subroutines, RandBinSearch and SizeReduction. In Section 4.4, we present the full proof of Theorem 4.1.

4.2 Random Binary Search

RandBinSearch is called by the function $\mathit{Advance}(1)$. It reduces the size of a candidate set from some value $v\leq n$ to some value $\ell<v$ in $\log v-\log\ell$ queries.

Lemma 4.4**.**

Let $x\in\{0,1\}^{n}$ with $f(x)=s$ and let $V$ be any set with $\pi(s+1)\in V$ and $\pi(j)\not\in V$ for $j\leq s$. Let $v:=|V|$ and $\ell\in\mathbb{N}$ with $\ell<v$. Algorithm 4.3 reduces the size of $V$ to $\ell$ using at most $\lceil\log v-\log\ell\rceil$ queries.

Proof.

Since $f(x)=s$, we have $x_{\pi(i)}=z_{\pi(i)}$ for all $i\in[s]$ and $x_{\pi(s+1)}\not=z_{\pi(s+1)}$. Also $\pi(s+1)\in V$ and $\pi(j)\not\in V$ for $j\leq s$. Therefore, either we have $f(y^{\prime})>s$ in line 4.3 or we have $f(y^{\prime})=s$. In the former case, the bit $\pi(s+1)$ was flipped, and hence $\pi(s+1)\in F$ must hold. In the latter case the bit in position $\pi(s+1)$ bit was not flipped and we infer $\pi(s+1)\not\in F$.

The runtime bound follows from the fact that the size of the set $V$ halves in each iteration. ∎

We call RandBinSearch in Algorithm 4.2 (line 4.2) to reduce the size of $V_{s+1}$ to $n/\alpha_{1}^{d}$, or, put differently, to reduce the number of candidates for $\pi(s+1)$ to $n/\alpha_{1}^{d}$. As the initial size of $V_{s+1}$ is at most $n$, this requires at most $d\log\alpha_{1}$ queries by Lemma 4.4.

Lemma 4.5**.**

A call of $\mathit{Advance}(1)$ in Algorithm 4.2 requires at most $\alpha_{1}+\alpha_{1}d\log\alpha_{1}$ queries.

Proof.

The two occasions where queries are made in Algorithm 4.2 are in line 4.2 and in line 4.2. Line 4.2 is executed at most $\alpha_{1}$ times, each time causing exactly one query. Each call to RandBinSearch in line 4.2 causes at most $d\log\alpha_{1}$ queries, and RandBinSearch is called at most $\alpha_{1}$ times. ∎

4.3 Size Reduction

We describe the second subroutine of Algorithm 4.2, SizeReduction. This routine is used to reduce the sizes of the up to $\alpha_{\ell-1}$ candidate sets returned by a recursive call $\mathit{Advance}(\ell-1)$ from some value $\leq n/\alpha_{\ell-1}^{d}$ to at most the target size of level $\ell$, which is $n/{\alpha_{\ell}^{d}}$. As we shall see below, this requires an expected number of $O(1)\alpha_{\ell-1}d(\log\alpha_{\ell}-\log\alpha_{\ell-1})/\log\alpha_{\ell-1}$ queries. The pseudo-code of SizeReduction is given in Algorithm 4.6. It repeatedly calls a subroutine ReductionStep that reduces the sizes of at most $k$ candidate sets to a $k$th fraction of their original size using at most $O(k)$ queries, where $k$ is a parameter. We use ReductionStep with parameter $k=\alpha_{\ell-1}$ repeatedly to achieve the full reduction of the sizes to at most $n/\alpha_{\ell}^{d}$.

ReductionStep is given a set $J$ of at most $k$ indices and a string $x$ with $f(x)\geq\max J$. The goal is to reduce the size of each candidate set $V_{j}$, $j\in J$, below a target size $m$ where $m\geq|V_{j}|/k$ for all $j\in J$. The routine works in phases of several iterations each. Let $J$ be the set of indices of the candidate sets that are still above the target size at the beginning of an iteration. For each $j\in J$, we randomly choose a subset $F_{j}\subseteq V_{j}$ of size $|V_{j}|/k$. We create a new bit-string $y^{\prime}$ from $x$ by flipping the entries in positions $\bigcup_{j\in J}F_{j}$. Since the sets $V_{j}$, $j\leq s=\max J$, are pairwise disjoint, we have either $f(y^{\prime})\geq\max J$ or $f(y^{\prime})=j-1$ for some $j\in J$. In the first case, i.e., if $f(y^{\prime})\geq\max J$, none of the sets $V_{j}$ was hit, and for all $j\in J$ we can remove the subset $F_{j}$ from the candidate set $V_{j}$. We call such queries “off-trials”. An off-trial reduces the size of all sets $V_{j}$, $j\in J$, to a $(1-1/k)$th fraction of their original size. If, on the other hand, we have $f(y^{\prime})=j-1$ for some $j\in J$, we can replace $V_{j}$ by set $F_{j}$ as $\pi(j)\in F_{j}$ must hold. Since $|F_{j}|=|V_{j}|/k\leq m$ by assumption, this set has now been reduced to its target size and we can remove it from $J$.

We continue in this way until at least half of the indices are removed from $J$ and at least $ck$ off-trials occurred, for some constant $c$ satisfying $(1-1/k)^{ck}\leq 1/2$. We then proceed to the next phase. Consider any $j$ that is still in $J$. The size of $V_{j}$ was reduced by a factor $(1-1/k)$ at least $ck$ times. Thus its size was reduced to at most half its original size. We may thus halve $k$ without destroying the invariant $m\geq|V_{j}|/k$ for $j\in J$. The effect of halving $k$ is that the relative size of the sets $F_{j}$ will be doubled for the sets $V_{j}$ that still take part in the reduction process.

Lemma 4.8**.**

Let $k\in\mathbb{N}$ and let $J\subseteq[n]$ be a set of at most $k$ indices with $s=\max J$. Assume that invariants (1) and (4) hold. Let $x\in\{0,1\}^{n}$ be such that $f(x)\geq\max J$ and let $m\in\mathbb{N}$ be such that $m\geq|V_{j}|/k$ for all $j\in J$. In expectation it takes $O(k)$ queries until ${\texttt{ReductionStep}}(k,J,m,x)$ has reduced the size of $V_{j}$ to at most $m$ for each $j\in J$.

Proof.

Let $c$ be some constant. We show below that—for a suitable choice of $c$—after an expected number of at most $ck$ queries both conditions in line 4.7 are satisfied. Assuming this to hold, we can bound the total expected number of queries until the size of each of the sets $V_{j}$ has been reduced to $m$ by

[TABLE]

as desired.

In each iteration of the repeat-loop we either hit an index in $J$ and hence remove it from $J$ or we have an off-trial. The probability of an off-trial is at least $(1-1/k)^{k}$ since $|J|\leq k$ always. Thus the probability of an off-trial is at least $(2e)^{-1}$ and hence the condition $o\geq ck$ holds after an expected number of $O(k)$ iterations.

As long as $|J|\geq\ell/2$, the probability of an off-trial is at most $(1-1/k)^{\ell/2}$ and hence the probability that a set is hit is at least $1-(1-1/k)^{\ell/2}$. Since $\ln(1-1/k)\leq-1/k$ we have $(1-1/k)^{\ell/2}=\exp(\ell/2\ln(1-1/k))\leq\exp(-\ell/(2k))$ and hence $1-(1-1/k)^{\ell/2}\geq 1-\exp(-\ell/(2k))\geq\ell/(2k)$. Thus the expected number of iterations to achieve $\ell/2$ hits is $O(k)$.

If a candidate set $V_{j}$ is hit in the repeat-loop, its size is reduced to $|V_{j}|/k$. By assumption, this is bounded by $m$. If $V_{j}$ is never hit, its size is reduced at least $ck$ times by a factor $(1-1/k)$. By choice of $c$, its size at the end of the phase is therefore at most half of its original size. Thus after replacing $k$ by $k/2$ we still have $|V_{j}|/k\leq m$ for $j\in J$. ∎

It is now easy to determine the complexity of SizeReduction.

Corollary 4.9**.**

Let $k\in\mathbb{N}$, $J$, and $y$ be as in Lemma 4.8. Let further $d\in\mathbb{N}$ and $\alpha\in\mathbb{R}$ such that $\max_{j\in J}|V_{j}|=n/\alpha^{d}$. Let $\beta\in\mathbb{R}$ with $\beta>\alpha$.

Using at most $d(\log\beta-\log\alpha)/\log k$ calls to Algorithm 4.7 we can reduce the maximal size $\max_{j\in J}|V_{j}|$ to $n/\beta^{d}$. The overall expected number of queries needed by SizeReduction to achieve this reduction is $O(1)kd(\log\beta-\log\alpha)/\log k$.

Proof.

The successive calls can be done as follows. We first call $\texttt{ReductionStep}(k,J,n/(k\alpha^{d}),x)$. By Lemma 4.8 it takes an expected number of $O(k)$ queries until the algorithm terminates. The sets $V_{j}$, $j\in J$, now have size at most $n/(k\alpha^{d})$. We next call $\texttt{ReductionStep}(k,J,n/(k^{2}\alpha^{d}),x)$. After the $h$th such call we are left with sets of size at most $n/(k^{h}\alpha^{d})$. For $h=d(\log\beta-\log\alpha)/\log k$ we have $k^{h}\geq(y/\alpha)^{d}$. The total expected number of queries at this point is $O(1)kd(\log\beta-\log\alpha)/\log k$. ∎

4.4 Proof of Theorem 4.1

It remains to show that the first phase of Algorithm 4.2 takes $O(n\log\log n)$ queries in expectation.

Theorem 4.10**.**

Let $q\in n-\Theta(n/\log n)$. Algorithm 4.2 identifies positions $\pi(1),\ldots,\pi(q)$ and the corresponding entries $z_{\pi(1)},\ldots,z_{\pi(q)}$ of $z$ in these positions using an expected number of $O(n\log\log n)$ queries.

We recall from the beginning of Section 4 that Theorem 4.1 follows from Theorem 4.10 by observing that once the $q\in n-\Theta(n/\log n)$ first positions of $z$ have been identified, the remaining $\Theta(n/\log n)$ values $z_{\pi(q+1)},\ldots,z_{\pi(n)}$ can be identified by binary search in expected linear time (already a naïve implementation of binary search finds each $z_{\pi(j)}$ in $O(\log n)$ queries, as demonstrated in Section 3).

To prove Theorem 4.10, we bound by $O(n\log\log n)$ the expected number of queries needed if no failure in any call of $\mathit{Advance}$ happened and by $O(n)$ the expected number of queries caused by failures (the latter is postponed to Section 4.5).

If no failure in any call of $\mathit{Advance}$ happened, the expected number of queries is bounded by

[TABLE]

where $c$ is the constant hidden in the $O(1)$-term in Lemma 4.8. To verify this formula, observe that we fill the $(i-1)$st level $\alpha_{i}/\alpha_{i-1}$ times before level $i$ has reached its capacity of $\alpha_{i}$ candidate sets. To add $\alpha_{i-1}$ candidate sets from level $i-1$ to level $i$, we need to reduce their sizes from $n/\alpha_{i-1}^{d}$ to $n/\alpha_{i}^{d}$. By Corollary 4.9 this requires at most $\alpha_{i-1}dc(\log\alpha_{i}-\log\alpha_{i-1})/\log\alpha_{i-1}$ queries. The additional $\alpha_{1}d\log\alpha_{1}$ term accounts for the queries needed to move the sets from level [math] to level $1$; i.e., for the randomized binary search algorithm through which we initially reduce the sizes of the sets $V_{i}$ to $n/\alpha_{1}^{d}$—requiring at most $d\log\alpha_{1}$ queries per call. Finally, the term $\alpha_{t}\log n/\log\alpha_{t}$ accounts for the final reduction of the sets $V_{i}$ to a set containing only one single element (at this stage we shall finally have $V_{i}=\{\pi(i)\}$). More precisely, this term is $\left(\alpha_{t}(\log n-d\log\alpha_{t})\right)/\log\alpha_{t}$ but we settle for upper bounding this expression by the term given in the formula.

Next we need to bound the number of queries caused by failures. We show that, on average, not too many failures happen. More precisely, we will show in Corollary 4.13 that the expected number of level-$i$ failures is at most $n^{2}/((n-q)(\alpha_{i}^{d-1}-1))$. By Corollary 4.9, each such level-$i$ failure causes an additional number of at most $1+\alpha_{i}dc(\log\alpha_{i+1}-\log\alpha_{i})/\log\alpha_{i}$ queries (the $1$ counts for the query through which we discover that $\pi(s+1)\in V_{1}\cup\ldots\cup V_{s}$). Thus,

[TABLE]

bounds the expected number of additional queries caused by failures.

Recall the choice of $t$ and the capacities $\alpha_{i}$. We have $\alpha_{1}=\log n$ and $\alpha_{i}=\alpha_{i-1}^{2}=(\log n)^{2^{i-1}}$; $t$ is maximal such that $\alpha_{t}^{d}\leq n$. Then $\alpha_{t}\geq\sqrt{n}$ and hence $\log\alpha_{t}=\Omega(\log n)$. The parameter $d$ is any constant that is at least $4$. With these parameter settings, formula (1) evaluates to

[TABLE]

and, somewhat wastefully, we can bound formula (2) from above by

[TABLE]

where the first equation is by construction of the values of $\alpha_{i}$, the inequality uses the fact that the geometric sum is dominated by the first term, and the last equality stems from our choice $\alpha_{1}=\log n$ and $d\geq 4$. This shows, once Corollary 4.13 is proven, that the overall expected number of queries sums to $O(n\log\log n)+O(n)=O(n\log\log n)$. ∎

4.5 Failures

We derive a bound on the expected number of failures. We do so in two steps. We first bound the worst-case number of calls of $\mathit{Advance}(\ell)$ for any fixed $\ell$ and then bound the probability that any particular call may fail.

In order to bound the worst-case number of calls of $\mathit{Advance}(\ell)$, we observe that any such call returns a non-empty set and distinct calls return disjoint sets. Thus there can be at most $q$ calls to $\mathit{Advance}(\ell)$ for any $\ell$. Before a call to $\mathit{Advance}(\ell)$, we have $\pi(s+1)\not\in\bigcup_{j\leq s}V_{j}$, and hence any call increases $s$ by at least 1. This is obvious for $\mathit{Advance}(1)$ and holds for $\ell>1$, since $\mathit{Advance}(\ell)$ immediately calls $\mathit{Advance}(\ell-1)$.

We turn to the probabilistic part of the analysis. Key to the failure analysis is the following observation.

Lemma 4.11**.**

Each $V_{j}$ has the property that $V_{j}\setminus\{\pi(j)\}$ is a random subset of $V_{j}^{*}\setminus\{\pi(j)\}$ of size $|V_{j}|-1$, where $V_{j}^{*}$ is as defined in line 4.2 of Algorithm 4.2.

Proof.

$V_{j}$ is initialized to $V_{j}^{*}$; thus the claim is true initially. In RandBinSearch, a random subset $F$ of $V_{j}$ is chosen and $V_{j}$ is reduced to $F$ (if $\pi(j)\in F)$ or to $V_{j}\setminus F$ (if $\pi(j)\not\in F)$. In either case, the claim stays true. The same reasoning applies to ReductionStep. ∎

Lemma 4.12**.**

Let $q\in n-\Theta(n/\log n)$ be the number of indices $i$ for which we determine $\pi(i)$ and $z_{\pi(i)}$ in the first phase. The probability that any particular call of $\mathit{Advance}(\ell)$ fails is less than $n(n-q)^{-1}(\alpha_{\ell}^{d-1}-1)^{-1}$.

Proof.

A failure happens if both $x$ and $y=x\oplus 1_{[n]\setminus\bigcup_{i=1}^{s}{V_{i}}}$ have a score of more than $s$ or equal to $s$ in line 4.2 of Algorithm 4.2. This is equivalent to $\pi(s+1)\in\bigcup_{i=1}^{s}{V_{i}}$.

Let $k$ be the number of indices $i\in[n]$ for which $V_{i}$ is on the last level; i.e., $k:=|\{i\in[n]\mid|V_{i}|=1\}|$ is the number of sets $V_{i}$ which have been reduced to singletons already. Note that these sets satisfy $V_{i}=\{\pi(i)\}$. Therefore, they cannot contain $\pi(s+1)$ and we do not need to take them into account.

A failure on level $\ell$ occurs only if $\pi(s+1)\in\bigcup_{j=1}^{s}{V_{j}}$ and the size of each candidate set $V_{1},\ldots,V_{s}$ has been reduced already to at most $n/\alpha_{\ell}^{d}$. There are at most $\alpha_{j}$ candidate sets on each level $j\geq\ell$. By construction, the size of each candidate set on level $j$ is at most $n/\alpha_{j}^{d}$. By Lemma 4.11, the probability that $\pi(s+1)\in\bigcup_{j=1}^{s}{V_{j}}$ is at most

[TABLE]

By definition we have $\alpha_{j}\geq\alpha_{\ell}^{(2^{j-\ell})}$ and in particular we have $\alpha_{j}\geq\alpha_{\ell}^{{j-\ell}}$. Therefore expression (3) can be bounded from above by

[TABLE]

∎

Since $\mathit{Advance}(\ell)$ is called at most $q$ times we immediately get the following.

Corollary 4.13**.**

Let $\ell\in[t]$. The expected number of level $\ell$ failures is less than

[TABLE]

5 The Lower Bound

We prove a tight lower bound for the randomized query complexity of the HiddenPermutation problem. The lower bound is stated in the following:

Theorem 5.1**.**

The randomized query complexity of the HiddenPermutation problem with $n$ positions is $\Omega(n\log\log n)$.

To prove a lower bound for randomized query schemes, we appeal to Yao’s principle. That is, we first define a hard distribution over the secrets and show that every deterministic query scheme for this hard distribution needs $\Omega(n\log\log n)$ queries in expectation. This part of the proof is done using a potential function argument.

Hard Distribution.

Let $\Pi$ be a permutation drawn uniformly among all the permutations of $[n]$ (in this section, we use capital letters to denote random variables). Given such a permutation, we let our target string $Z$ be the one satisfying $Z_{\Pi(i)}=i(\textrm{ mod }2)$ for $i=1,\dots,n$. Since $Z$ is uniquely determined by the permutation $\Pi$, we will mostly ignore the role of $Z$ in the rest of this section. Finally, we use $F(x)$ to denote the value of the random variable $f_{Z,\Pi}(x)$ for $x\in\left\{0,1\right\}^{n}$. We will also use the notation $a\equiv b$ to mean that $a\equiv b(\textrm{ mod }2)$.

Deterministic Query Schemes.

By fixing the random coins, a randomized solution with expected $t$ queries implies the existence of a deterministic query scheme with expected $t$ queries over our hard distribution. The rest of this section is devoted to lower bounding $t$ for such a deterministic query scheme.

Recall that a deterministic query scheme is a decision tree $T$ in which each node $v$ is labeled with a string $x_{v}\in\{0,1\}^{n}$. Each node has $n+1$ children, numbered from 0 to $n$, and the $i$th child is traversed if $F(x_{v})=i$. To guarantee correctness, no two inputs can end up in the same leaf.

For a node $v$ in the decision tree $T$, we define $\max_{v}$ as the largest value of $F$ seen along the edges from the root to $v$. Note that $\max_{v}$ is not a random variable. Also, we define $S_{v}$ as the subset of inputs (as outlined above) that reach node $v$.

Update Rules for Candidate Sets.

We use a potential function which measures how much “information” the queries asked have revealed about $\Pi$. Our goal is to show that the expected increase in the potential function after asking each query is small. Our potential function depends crucially on the candidate sets. The update rules for the candidate sets are slightly more specific than the ones in Section 2 because we now have a fixed connection between the two parts of the secret. We denote the candidate set for $\pi(i)$ at node $v$ with $V_{i}^{v}$. At the root node $r$, we have $V^{r}_{i}=[n]$ for all $i$. Let $v$ be a node in the tree and let $w_{0},\dots,w_{n}$ be its children ($w_{i}$ is traversed when the score $i$ is returned). Let $P_{0}^{v}$ (resp. $P_{1}^{v}$) be the set of positions in $x_{v}$ that contain 0 (resp. 1). Thus, formally, $P_{0}^{v}=\{i\mid x_{v}[i]=0\}$ and $P_{1}^{v}=\{i\mid x_{v}[i]=1\}$.222To prevent our notations from becoming too overloaded, here and in the remainder of the section we write $x=(x[1],\ldots,x[n])$ instead of $x=(x_{1},\ldots,x_{n})$ The precise definition of candidate sets is as follows:

[TABLE]

As with the upper bound case, the candidate sets have some very useful properties. These properties are slightly different from the ones observed before, due to the fact that some extra information has been announced to the query algorithm. We say that a candidate set $V_{i}^{v}$ is active (at $v$) if the following conditions are met: (i) at some ancestor node $u$ of $v$, we have $F(x_{u})=i-1$, (ii) at every ancestor node $w$ of $u$ we have $F(x_{w})<i-1$, and (iii) $i<\min\left\{n/3,\max_{v}\right\}$. We call $V_{\max_{v}+1}^{v}$ pseudo-active (at $v$).

For intuition on the requirement $i<n/3$, observe from the following lemma that $V^{v}_{\max_{v}+1}$ contains all sets $V^{v}_{i}$ for $i\leq\max_{v}$ and $i\equiv\max_{v}$. At a high level, this means that the distribution of $\Pi(\max_{v}+1)$ is not independent of $\Pi(i)$ for $i\equiv\max_{v}$. The bound $i<n/3$, however, forces the dependence to be rather small (there are not too many such sets). This greatly helps in the potential function analysis.

We prove the following lemma with arguments similar to those in the proof of Theorem 2.1.

Lemma 5.2**.**

The candidate sets have the following properties:

(i) Two candidate sets $V_{i}^{v}$ and $V_{j}^{v}$ with $i<j\leq\max_{v}$ and $i\not\equiv j$ are disjoint.

(ii) An active candidate set $V_{j}^{v}$ is disjoint from any candidate set $V_{i}^{v}$ provided $i<j<\max_{v}$.

(iii) The candidate set $V_{i}^{v}$, $i\leq\max_{v}$ is contained in the set $V_{\max_{v}+1}^{v}$ if $i\equiv\max_{v}$ and is disjoint from it if $i\not\equiv\max_{v}$.

(iv) For two candidate sets $V_{i}^{v}$ and $V_{j}^{v}$, $i<j$, if $V_{i}^{v}\cap V_{j}^{v}\not=\emptyset$ then $V_{i}^{v}\subset V_{j}^{v}$.

Proof.

Let $w$ be the ancestor of $v$ where the function returns score $\max_{v}$.

To prove (i), observe that in $w$, one of $V_{i}^{w}$ and $V_{j}^{w}$ is intersected with $P_{0}^{w}$ while the other is intersected with $P_{1}^{w}$ and thus they are made disjoint.

To prove (ii), we can assume $i\equiv j$ as otherwise the result follows from the previous case. Let $u$ be the ancestor of $v$ such that $F(x_{u})=j-1$ and that in any ancestor of $u$, the score returned by the function is smaller than $j-1$. At $u$, $V_{j}^{u}$ is intersected with $P_{j-1\bmod 2}^{u}$ while $V_{i}^{v}$ is intersected with $P_{i\bmod 2}^{u}$. Since $i\equiv j$, it follows that they are again disjoint.

For (iii), the latter part follows as in (i). Consider an ancestor $v^{\prime}$ of $v$ and let $w_{j}$ be the $j$th child of $v^{\prime}$ that is also an ancestor of $v$. We use induction and we assume $V_{i}^{v^{\prime}}\subset V_{\max_{v}+1}^{v^{\prime}}$. If $j<\max_{v}$, then $V_{\max_{v}+1}^{v^{\prime}}=V_{\max_{v}+1}^{w_{j}}$ which means $V_{i}^{w_{j}}\subset V_{\max_{v}+1}^{w_{j}}$. If $j=\max_{v}$, then $V_{\max_{v}+1}^{w_{j}}=V_{\max_{v}+1}^{v^{\prime}}\cap P_{\max_{v}(\textrm{ mod }2)}^{v^{\prime}}$ and notice that in this case also $V_{i}^{w_{j}}=V_{i}^{v^{\prime}}\cap P_{i(\textrm{ mod }2)}^{v^{\prime}}$ which still implies $V_{i}^{w_{j}}\subset V_{\max_{v}+1}^{w_{j}}$.

To prove (iv), first observe that the statement is trivial if $i\not\equiv j$. Also, if the function returns score $j-1$ at any ancestor of $v$, then by the same argument used in (ii) it is possible to show that $V_{i}^{v}\cap V_{j}^{v}=\emptyset$. Thus assume $i\equiv j$ and the function never returns value $j-1$. In this case, it is easy to see that an inductive argument similar to (iii) proves that $V_{i}^{v^{\prime}}\subset V_{j}^{v^{\prime}}$ for every ancestor $v^{\prime}$ of $v$. ∎

Corollary 5.3**.**

Every two distinct active candidate sets $V_{i}^{v}$ and $V_{j}^{v}$ are disjoint.

Remember that the permutation $\Pi$ was chosen uniformly and randomly. Soon, we shall see that this fact combined with the above properties imply that $\Pi(i)$ is uniformly distributed in $V_{i}^{v}$, when $V_{i}^{v}$ is active. The following lemma is needed to prove this.

Lemma 5.4**.**

Consider a candidate set $V_{i}^{v}$ and let $i_{1}<\cdots<i_{k}<i$ be the indices of candidate sets that are subsets of $V_{i}^{v}$. Let $\sigma:=(\sigma_{1},\cdots,\sigma_{i})$ be a sequence without repetition from $[n]$ and let $\sigma^{\prime}:=(\sigma_{1},\cdots,\sigma_{i-1})$. Let $n_{\sigma}$ and $n_{\sigma^{\prime}}$ be the number of permutations in $S_{v}$ that have $\sigma$ and $\sigma^{\prime}$ as a prefix, respectively. If $n_{\sigma}>0$, then $n_{\sigma^{\prime}}=(|V_{i}^{v}|-k)n_{\sigma}$.

Proof.

Consider a permutation $\pi\in S_{v}$ that has $\sigma$ as a prefix. This implies $\pi(i)\in V_{i}^{v}$. For an element $s\in V_{i}^{v}$, $s\not=i_{j}$, $1\leq j\leq k$, let $\pi_{s}$ be the permutation obtained from $\pi$ by placing $s$ at position $i$ and placing $\pi(i)$ where $s$ used to be. Since $s\not=i_{j}$, $1\leq j\leq k$, it follows that $\pi_{s}$ has $\sigma^{\prime}$ as prefix and since $s\in V_{i}^{v}$ it follows that $\pi_{s}\in S_{v}$. It is easy to see that for every permutation in $S_{v}$ that has $\sigma$ as a prefix we will create $|V_{i}^{v}|-k$ different permutations that have $\sigma^{\prime}$ as a prefix and all these permutations will be distinct. Thus, $n_{\sigma^{\prime}}=(|V_{i}^{v}|-k)n_{\sigma}$. ∎

Corollary 5.5**.**

Consider a candidate set $V_{i}^{v}$ and let $i_{1}<\cdots<i_{k}<i$ be the indices of candidate sets that are subsets of $V_{i}^{v}$. Let $\sigma^{\prime}:=(\sigma_{1},\cdots,\sigma_{i-1})$ be a sequence without repetition from $[n]$ and let $\sigma_{1}:=(\sigma_{1},\cdots,\sigma_{i-1},s_{1})$ and $\sigma_{2}:=(\sigma_{1},\cdots,\sigma_{i-1},s_{2})$ in which $s_{1},s_{2}\in V_{i}^{v}$. Let $n_{\sigma_{1}}$ and $n_{\sigma_{2}}$ be the number of permutations in $S_{v}$ that have $\sigma_{1}$ and $\sigma_{2}$ as a prefix, respectively. If $n_{\sigma_{1}},n_{\sigma_{2}}>0$, then $n_{\sigma_{1}}=n_{\sigma_{2}}$.

Proof.

Consider a sequence $s_{1},\cdots,s_{i}$ without repetition from $[n]$ such that $s_{j}\in V_{j}^{v}$, $1\leq j\leq i$. By the previous lemma $\Pr[\Pi(1)=s_{1}\wedge\cdots\wedge\Pi(i-1)=s_{i-1}\wedge\Pi(i)=s_{i}]=\Pr[\Pi(1)=s_{1}\wedge\cdots\wedge\Pi(i-1)=s_{i-1}]\cdot(1/|V_{i}^{v}|)$. ∎

Corollary 5.6**.**

If $V_{i}^{v}$ is active, then we have:

• (i)

$\Pi(i)$* is independent of $\Pi(1),\cdots,\Pi(i-1)$.*

• (ii)

$\Pi(i)$* is uniformly distributed in $V_{i}^{v}$.*

5.1 The Potential Function

We define the potential of an active candidate set $V_{i}^{v}$ as $\log\log{(2n/|V_{i}^{v}|)}$. This is inspired by the upper bound: a potential increase of $1$ corresponds to a candidate set advancing one level in the upper bound context (in the beginning, a set $V_{i}^{v}$ has size $n$ and thus its potential is 0 while at the end its potential is $\Theta(\log\log n)$. With each level, the quantity $n$ divided by the size of $V_{i}$ is squared). We define the potential at a node $v$ as

[TABLE]

in which $A_{v}$ is the set of indices of active candidate sets at $v$ and $\operatorname{Con}_{v}$ is the number of candidate sets contained inside $V_{\max_{v}+1}^{v}$. Note that from Lemma 5.2, it follows that $\operatorname{Con}_{v}=\lfloor\max_{v}/2\rfloor$.

The intuition for including the term $\operatorname{Con}_{v}$ is the same as our requirement $i<n/3$ in the definition of active candidate sets, namely that once $\operatorname{Con}_{v}$ approaches $|V_{\max_{v}+1}^{v}|$, the distribution of $\Pi(\max_{v}+1)$ starts depending heavily on the candidate sets $V_{i}^{v}$ for $i\leq\max_{v}$ and $i\equiv\max_{v}$. Thus we have in some sense determined $\Pi(\max_{v}+1)$ already when $|V_{\max_{v}+1}^{v}|$ approaches $\operatorname{Con}_{v}$. Therefore, we have to take this into account in the potential function since otherwise changing $V_{\max_{v}+1}^{v}$ from being pseudo-active to being active could give a huge potential increase.

The following is the main lemma that we wish to prove; it tells us that the expected increase of the potential function after each query is constant. The proof of this lemma is the main part of the proof of Theorem 5.1.

Lemma 5.7**.**

Let $v$ be a node in $T$ and let ${\textbf{i}}_{v}$ be the random variable giving the value of $F(x_{v})$ when $\Pi\in S_{v}$ and [math] otherwise. Also let $w_{0},\dots,w_{n}$ denote the children of $v$, where $w_{j}$ is the child reached when $F(x_{v})=j$. Then, $\operatorname{\mathbb{E}}[\varphi(w_{{\textbf{i}}_{v}})-\varphi(v)\mid\Pi\in S_{v}]=O(1)$.

Note that we have $\operatorname{\mathbb{E}}[\varphi(w_{{\textbf{i}}_{v}})-\varphi(v)\mid\Pi\in S_{v}]=\sum_{a=0}^{n}\Pr[F(x_{v})=a\mid\Pi\in S_{v}](\varphi(w_{a})-\varphi(v)).$ We consider two main cases: $F(x_{v})\leq\max_{v}$ and $F(x_{v})>\max_{v}$. In the first case, the maximum score will not increase in $w_{a}$ which means $w_{a}$ will have the same set of active candidate sets. In the second case, the pseudo-active candidate set $V_{\max_{v}+1}^{v}$ will turn into an active set $V_{\max_{v}+1}^{w_{a}}$ at $w_{a}$ and $w_{a}$ will have a new pseudo-active set. While this second case looks more complicated, it is in fact the less critical part of the analysis. This is because the probability of suddenly increasing the score by a large $\alpha$ is extremely small (we will show that it is roughly $2^{-\Omega(\alpha)}$) which subsumes any significant potential increase for values of $a>\max_{v}$.

Let $a_{1},\dots,a_{|A_{v}|}$ be the indices of active candidate sets at $v$ sorted in increasing order. We also define $a_{|A_{v}|+1}=\max_{v}+1$. For a candidate set $V_{i}^{v}$, and a Boolean $b\in\left\{0,1\right\}$, let $V_{i}^{v}(b)=\left\{j\in V_{i}^{v}\mid x_{v}[j]=b\right\}$. Clearly, $|V_{i}^{v}(0)|+|V_{i}^{v}(1)|=|V_{i}^{v}|$. For even $a_{i}$, $1\leq i\leq|A_{v}|$, let

[TABLE]

and for odd $i$, let

[TABLE]

Thus $\varepsilon_{i}$ is the fraction of locations in $V_{i}^{v}$ that contain values that does not match $Z_{\Pi(i)}$. Also, we define

[TABLE]

Note that $\varepsilon^{\prime}_{i}=0$ if $a_{i+1}=a_{i}+1$.

With these definitions, it is clear that we have

[TABLE]

The important fact is that we can also bound other probabilities using the values of the $\varepsilon_{i}$ and $\varepsilon^{\prime}_{i}$: We can show that (details follow)

[TABLE]

and

[TABLE]

Thus we have bounds on the changes in the size of the active candidate sets in terms of the values $\varepsilon_{i}$. The probability of making the various changes is also determined from the values $\varepsilon_{i}$ and $\varepsilon_{i}^{\prime}$ and finally the potential function is defined in terms of the sizes of these active candidate sets. Thus proving Lemma 5.7 reduces to proving an inequality showing that any possible choice of the values for the $\varepsilon_{i}$ and $\varepsilon^{\prime}_{i}$ provide only little expected increase in potential.

Proof Sketch.

Since the full calculations are rather lengthy, in the following paragraphs we will provide the heart of the analysis by making simplifying assumptions that side-step some uninteresting technical difficulties that are needed for the full proof. We assume that all values $\varepsilon^{\prime}_{i}$ are [math] or in other words, $a_{i}=i$ for all $i\leq\max_{v}$. Also, we ignore the term in $\varphi(v)$ involving $\operatorname{Con}_{v}$ and consider only the case where the score returned is no larger than $\max_{v}$ and $\max_{v}=n/4$.

Thus the expected increase in potential provided by the cases where the score does not increase is bounded by

[TABLE]

Also, we get that when a score of $j$ is returned, we update

[TABLE]

Examining $\varphi(w_{j})-\varphi(v)$ and the changes to the candidate sets, we get that there is one candidate set whose size decreases by a factor $\varepsilon_{j}$, and there are $j$ sets that change by a factor $1-\varepsilon_{j}$. Here we only consider the potential change caused by the sets changing by a factor of $\varepsilon_{j}$. This change is bounded by

[TABLE]

where we used $\log(1+x)\leq x$ for $x>0$ for the last inequality. The function $\varepsilon_{j}\rightarrow(1/\varepsilon_{j})^{\varepsilon_{j}}$ is decreasing for $0<\varepsilon\leq 1$. Also, if $\varepsilon>0$ then $\varepsilon\geq 1/n$ by definition of $\varepsilon$. We can therefore upper bound the sum by setting $\varepsilon_{j}=4/n$ for all $j$. To continue these calculations, we use Lemma 5.2 to conclude that the active candidate sets are disjoint and hence the sum of their sizes is bounded by $n$. We now divide the sum into summations over indices where $|V_{j}^{v}|$ is in the range $[2^{i}:2^{i+1}]$ (there are at most $n/2^{i}$ such indices):

[TABLE]

Now the sum over the terms where $i>\log\log n$ is clearly bounded by a constant since the $2^{i}$ in the denominator cancels the $\log(n/4)$ term and we get a geometric series of this form. For $i<\log\log n$, we have $\log(n/4)/\log(n/2^{i})=O(1)$ and we again have a geometric series summing to $O(1)$.

The full proof is very similar in spirit to the above, just significantly more involved due to the unknown values $\varepsilon_{i}$ and $\varepsilon_{i}^{\prime}$. The complete proof is given in the next section.

5.2 Formal Proof of Lemma 5.7

As we did in the proof sketch we write

[TABLE]

As discussed, we divide the above summation into two parts: one for $a\leq\max_{v}$ and another for $a>\max_{v}$:

[TABLE]

To bound the above two summations, it is clear that we need to handle $\Pr[F(x_{v})=a|\Pi\in S_{v}]$. In the next section, we will prove lemmas that will do this.

Bounding Probabilities

Let $a_{1},\dots,a_{|A_{v}|}$ be the indices of active candidate sets at $v$ sorted in increasing order. We also define $a_{|A_{v}|+1}=\max_{v}+1$. For a candidate set $V_{i}^{v}$, and a Boolean $b\in\left\{0,1\right\}$, let $V_{i}^{v}(b)=\left\{j\in V_{i}^{v}\mid x_{v}[j]=b\right\}$. Clearly, $|V_{i}^{v}(0)|+|V_{i}^{v}(1)|=|V_{i}^{v}|$. For even $a_{i}$, $1\leq i\leq|A_{v}|$, let $\varepsilon_{i}=|V_{i}^{v}(1)|/|V_{i}^{v}|$ and for odd $i$ let $\varepsilon_{i}=|V_{i}^{v}(0)|/|V_{i}^{v}|$. This definition might seem strange but is inspired by the following observation.

Lemma 5.8**.**

For $i\leq|A_{v}|$, $\Pr[F(x_{v})=a_{i}-1|\Pi\in S_{v}\wedge F(x_{v})>a_{i}-2]=\varepsilon_{i}$.

Proof.

Note that $F(x_{v})=a_{i}-1$ happens if and only if $F(x_{v})>a_{i}-2$ and $x_{v}[\Pi(a_{i})]\not\equiv a_{i}$. Since $V_{a_{i}}^{v}$ is an active candidate set, the lemma follows from Corollary 5.6 and the definition of $\varepsilon_{i}$. ∎

Let $\varepsilon^{\prime}_{i}:=\Pr[a_{i}\leq F(x_{v})<a_{i+1}-1|\Pi\in S_{v}\wedge F(x_{v})\geq a_{i}]$. Note that $\varepsilon^{\prime}_{i}=0$ if $a_{i+1}=a_{i}+1$.

Lemma 5.9**.**

*For $i\leq|A_{v}|$ we have $|V_{a_{i}}^{w_{j}}|=|V_{a_{i}}^{v}|$ for $j<a_{i}-1$, $|V_{a_{i}}^{w_{j}}|=\varepsilon_{i}|V_{a_{i}}^{v}|$ for $j=a_{i}-1$, and $|V_{a_{i}}^{w_{j}}|=(1-\varepsilon_{i})|V_{a_{i}}^{v}|$ for $j>a_{i}-1$. Also, *

[TABLE]

Proof.

Using Lemma 5.8, it is verified that

[TABLE]

Similarly, using the definition of $\varepsilon^{\prime}_{i}$ we can see that

[TABLE]

Using these equalities, we find that

[TABLE]

and

[TABLE]

Equality (7) follows from the above two equalities. To obtain Equality (8), we only need to consider the definition of $\varepsilon^{\prime}_{i}$ as an additional ingredient. The rest of the lemma follows directly from the definition of $\varepsilon_{i}$ and the candidate sets. ∎

Lemma 5.10**.**

*Let $b\in\left\{0,1\right\}$ be such that $b\equiv\max_{v}$ and let $k:=|V_{\max_{v}+1}^{v}(b)|$. Then, *

[TABLE]

Proof.

Conditioned on $F(x_{v})>\max_{v}-1$, $F(x_{v})$ will be equal to $\max_{v}$ if $x_{v}[\Pi(\max_{v}+1)]=b$. By definition, the number of positions in $V_{\max_{v}+1}^{v}$ that satisfy this is $k$. However, $V_{\max_{v}+1}^{v}$ contains $\operatorname{Con}_{v}$ candidate sets but since $V_{\max_{v}+1}^{v}$ can only contain a candidate set $V_{i}^{v}$ if $i\equiv\max_{v}$ (by Lemma 5.2), it follows from Lemma 5.4 that $\Pr[F(x_{v})=\max_{v}|\Pi\in S_{v}\wedge F(x_{v})>\max_{v}-1]=(k-\operatorname{Con}_{v})/(|V_{\max_{v}+1}^{v}|-\operatorname{Con}_{v})$. The lemma then follows from the previous lemma. ∎

Lemma 5.11**.**

Let $b\in\left\{0,1\right\}$ be such that $b\equiv\max_{v}$ and let $k:=|V_{\max_{v}+1}^{v}(b)|$. Then,

[TABLE]

Proof.

From the previous lemma we have that $\Pr[F(x_{v})=\max_{v}|\Pi\in S_{v}\wedge F(x_{v})>\max_{v}-1]=(k-\operatorname{Con}_{v})/(|V_{\max_{v}+1}^{v}|-\operatorname{Con}_{v})$. Thus,

[TABLE]

The claim now follows from Lemma 5.9. ∎

Remember that $P_{0}^{v}$ (resp. $P_{1}^{v}$) are the set of positions in $x_{v}$ that contain 0 (resp. 1).

Lemma 5.12**.**

*Let $x_{0}=|P_{0}^{v}|$ and $x_{1}=|P_{1}^{v}|$. Let $b_{i}$ be a Boolean such that $b_{i}\equiv i$. For $a\geq\max_{v}+2$ *

[TABLE]

Proof.

Notice that we have $V_{i}^{v}=[n]$ for $i\geq\max_{v}+2$ which means $i-1$ is the number of candidates sets $V_{j}^{v}$ contained in $V_{i}^{v}$ and among those $\lfloor i/2\rfloor$ are such that $i\equiv j$. Consider a particular prefix $\sigma=(\sigma_{1},\cdots,\sigma_{i-1})$ such that there exists a permutation $\pi\in S_{v}$ that has $\sigma$ as a prefix. This implies that $\sigma_{j}\in P_{b_{j}}^{v}$. Thus, it follows that there are $x_{b_{i}}-\lfloor i/2\rfloor$ elements $s\in P_{b_{i}}^{v}$ such that the sequences $(\sigma_{1},\cdots,\sigma_{i-1},s)$ can be the prefix of a permutation in $S_{v}$. Thus by Corollary 5.5, and for $i\geq\max_{v}+2$,

[TABLE]

and

[TABLE]

∎

Corollary 5.13**.**

*For $\max_{v}+1\leq a\leq n/3$ we have *

[TABLE]

Proof.

Since $x_{b_{i}}+x_{b_{i+1}}=n$, it follows that

[TABLE]

∎

Now we analyze the potential function. Remember that we have split the change in the potential function into two parts: one is captured by Equation (5) and the other by Equation (6). We claim each is bounded by a constant.

Lemma 5.14**.**

The value of Equation (5) is bounded by $O(1)$.

Proof.

We have,

[TABLE]

When $a\leq\max_{v}$ we have, $\max_{v}=\max_{w_{a}}$ and $\operatorname{Con}_{v}=\operatorname{Con}_{w_{a}}$. For $a<\max_{v}$, we also have $V^{w_{a}}_{\max_{v}+1}=V^{v}_{\max_{v}+1}$. It is clear from (9) that for $a_{i}\leq a<a_{i+1}-1$, all the values of $\varphi(w_{a})-\varphi(v)$ will be equal. Thus, (5) is equal to

[TABLE]

We now analyze each of these terms separately.

Analyzing (10)

We write (10) using (9) and Lemma 5.9(7). Using inequalities, $1-x\leq e^{-x}$, for $0\leq x\leq 1$, $\log(1+x)\leq x$ for $x\geq 0$, and $\sum_{1\leq i\leq k}y_{i}\log 1/y_{i}\leq Y\log(k/Y)$ for $y_{i}\geq 0$ and $Y=\sum_{1\leq i\leq k}y_{i}$, we obtain that (10) equals

[TABLE]

To bound (13), we use the fact that any two active candidate sets are disjoint. We break the summation into smaller chunks. Observe that $\prod_{j=1}^{i-1}(1-\varepsilon_{j})\leq e^{-\sum_{j=1}^{i-1}\varepsilon_{j}}$. Thus, let $J_{t}$, $t\geq 0$, be the set of indices such that for each $i\in J_{t}$ we have $2^{t}-1\leq\sum_{j=1}^{i-1}\varepsilon_{j}<2^{t+1}$. Now define $J_{t,k}=\left\{i\in J_{t}\mid n/2^{k+1}\leq|V_{a_{i}}^{v}|\leq n/2^{k}\right\}$, for $0\leq k\leq\log n$ and let $s_{t,k}=\sum_{i\in J_{t,k}}\varepsilon_{i}$. Observe that by the disjointness of two active candidate sets, $|J_{t,k}|\leq 2^{k+1}$. The sum (13) is thus equal to

[TABLE]

Now define $S_{t,r}=\sum_{2^{r-1}\leq k<2^{r}}s_{t,k}$. Remember that $\sum_{r=1}^{\log\log n}S_{t,r}<2^{t+1}$. (13) is thus at most

[TABLE]

To bound (14), define $J_{t}$ as before and let $p_{i}=\prod_{j=1}^{i-1}1/(1-\varepsilon_{j})$. Observe that the function $\log(1/(1-x))$ is a convex function which means if $s_{i}:=\sum_{j=1}^{i-1}\varepsilon_{j}$ is fixed, then $\prod_{j=1}^{i-1}1/(1-\varepsilon_{j})$ is minimized when $\varepsilon_{j}=s_{i}/(i-1)$. Thus,

[TABLE]

in which $j$ can be chosen to be any integer between 0 and $i-1$. We pick $j:=\max\left\{\lfloor s_{i}/8\rfloor,1\right\}$. Since $i\geq s_{i}$, we get for $i\in J_{t}$,

[TABLE]

Thus, we can write (14) as

[TABLE]

Analyzing ((11))

The analysis of this equation is very similar to (10). We can write (11) using (9) and (8). We also use the same technique as in analyzing (14). We find that the sum (11) equals

[TABLE]

Let $s_{i}:=\sum_{j=1}^{i}\varepsilon_{j}$, and $s^{\prime}_{i}:=\sum_{j=1}^{i-1}\varepsilon^{\prime}_{j}$. Similarly to the previous case, let $J_{t}$, $t\geq 0$, be the set of indices such that for each $i\in J_{t}$ we have $2^{t}-1\leq s_{i}+s^{\prime}_{i}\leq 2^{t+1}$. Also define $p_{i}=\prod_{j=1}^{i}1/(1-\varepsilon_{j})$ and $p^{\prime}_{i}=\prod_{j=1}^{i-1}1/(1-\varepsilon^{\prime}_{j})$. Using the previous techniques we can get that $p_{i}\geq 2^{s_{i}/8}$ and $p^{\prime}_{i}\geq 2^{s^{\prime}_{i}/8}$. We get that (11) is at most

[TABLE]

Analyzing (12)

Let $b\in\left\{0,1\right\}$ be such that $b\equiv\max_{v}$ and let $k:=|V_{\max_{v}+1}^{v}(b)|$. We use Lemma 5.10. Observe that we still have $\operatorname{Con}_{v}=\operatorname{Con}_{w_{\max_{v}}}$. Thus we can bound (12) from above by

[TABLE]

∎

We now return to the other remaining part of the change in the potential function.

Lemma 5.15**.**

The value of Equation (6) is bounded by $O(1)$.

Proof.

The big difference here is that the candidate set $V_{\max_{v}+1}^{w_{a}}$ becomes an active candidate set (if of course $\max_{v}+1<n/3$) at $w_{a}$ while $V_{\max_{v}+1}^{v}$ was not active at $v$. Because of this, we have

[TABLE]

Thus, we get that

[TABLE]

Using the previous ideas, it is easy to see that we can bound (17) from above by

[TABLE]

To analyze (16) by Lemma 5.11 we know that $\Pr[F(x_{v})>\max_{v}|\Pi\in S_{v}]\leq{|V_{\max_{v}+1}^{v}|-k\over|V_{\max_{v}+1}^{v}|-\operatorname{Con}_{v}}$ in which $k$ is as defined in the lemma. Note that in this case $|V_{\max_{v}+1}^{w_{a}}|=|V_{\max_{v}+1}^{v}|-k$. This implies that (16) is at most

[TABLE]

It is left to analyze (15). Let $x_{0}=|P_{0}^{v}|$ and $x_{1}=|P_{1}^{v}|$. Let $b_{i}$ be a Boolean such $b_{i}\equiv i$. Note that we have $|V_{\max_{v}+a+1}^{w_{\max_{v}+a}}|=x_{b_{{\max_{v}+a}}}=n-x_{b_{{\max_{v}+a+1}}}$. Using Corollary 5.13 we the term (15) is at most

[TABLE]

∎

As discussed earlier, to prove our main lemma, Lemma 5.7, we need to analyze the change in the potential function when going from node $v$ to a child of $v$. This change in potential was split into two terms, one captured by Equation (5) and another by Equation (6). By Lemmas 5.14 and 5.15, each of these terms is bounded by $O(1)$ and thus as a result, the change in the potential when going from $v$ to a child of $v$ is also bounded by $O(1)$.

To prove our lower bound, we simply need to observe that at the beginning, the potential is 0 since the algorithm essentially knows nothing. However, at the end, the sizes of all the candidates sets are small. Consider a node $v$ the decision where we have hit $\max_{v}=n/3$. By Lemma 5.2, it follows that $\Omega(n)$ candidate sets have $O(1)$ size. As a result, the value of the potential function at $v$ is $\Omega(n\log\log n)$ as each candidate set of size $O(1)$ contributes $\Omega(\log\log n)$ to the potential. However, as our main lemma, Lemma 5.7, shows, the expected increase in the potential function is $O(1)$. As a result, it follows that the expected depth of $v$ must be $\Omega(n\log\log n)$.

5.3 Concluding the Proof of Theorem 5.1

Intuitively, if the maximum score value increases after a query, it increases, in expectation, only by an additive constant. In fact, as shown in Corollary 5.13, the probability of increasing the maximum score value by $\alpha$ after one query is $2^{-\Omega(\alpha)}$. Thus, it follows from the definition of the active candidate sets that when the score reaches $n/3$ we expect $\Omega(n)$ active candidate sets. However, by Lemma 5.2, the active candidate sets are disjoint. This means that a fraction of them (again at least $\Omega(n)$ of them), must be small, or equivalently, their total potential is $\Omega(n\log\log n)$, meaning, at least $\Omega(n\log\log n)$ queries have been asked. In the rest of this section, we prove this intuition.

Given an input $(z,\pi)$, we say an edge $e$ in the decision tree $T$ is increasing if $e$ corresponds to an increase in the maximum score and it is traversed given the input $(z,\pi)$. We say that an increasing edge is short if it corresponds to an increase of at most $c$ in the maximum function score (in which $c$ is a sufficiently large constant) and we call it long otherwise. Let $N$ be the random variable denoting the number of increasing edges seen on input $\Pi$ before reaching a node with score greater than $n/3$. Let $L_{j}$ be the random indicator variable taking the value [math] if the $j$th increasing edge is short, and taking the value equal to the amount of increase in the score along this edge if not. If $j>N$, then we define $L_{j}=0$. Also let $W_{j}$ be the random variable corresponding to the node of the decision tree where the $j$th increase happens. As discussed, we have shown that for every node $v$, $\Pr[L_{j}\geq\alpha|W_{j}=v]\leq 2^{-\Omega(\alpha)}$. We want to upper bound $\sum_{j=1}^{n}\operatorname{\mathbb{E}}[L_{j}]$ (there are always at most $n$ increasing edges). From the above, we know that

[TABLE]

where the summation is taken over all nodes $v$ in the decision tree $T$. The computation shows $\sum_{j=1}^{n}\operatorname{\mathbb{E}}[L_{j}]\leq n/2^{\Omega(c)}$. By Markov’s inequality, we get that with probability at least $3/4$, we have $\sum_{j=1}^{n}L_{j}\leq n/2^{\Omega(c)}$. Thus, when the function score reaches $n/3$, short edges must account for $n/3-n/2^{\Omega(c)}$ of the increase which is at least $n/6$ for a large enough constant $c$. Since any short edge has length at most $c$, there must be at least $n/(6c)$ short edges. As discussed, this implies existence of $\Omega(n)$ active candidate sets that have size $O(1)$, meaning, their contribution to the potential function is $\Omega(\log\log n)$ each. We have thus shown:

Lemma 5.16**.**

Let $\ell$ be the random variable giving the leaf node of $T$ that the deterministic query scheme ends up in on input $\Pi$. We have $\varphi(\ell)=\Omega(n\log\log n)$ with probability at least $3/4$.

Finally, we show how Lemma 5.7 and Lemma 5.16 combine to give our lower bound. Essentially this boils down to showing that if the query scheme is too efficient, then the query asked at some node of $T$ increases the potential by $\omega(1)$ in expectation, contradicting Lemma 5.7. To show this explicitly, define t as the random variable giving the number of queries asked on input $\Pi$. We have $\operatorname{\mathbb{E}}[\textbf{t}]=t$, where $t$ was the expected number of queries needed for the deterministic query scheme. Also let $\ell_{1},\dots,\ell_{4t}$ be the random variables giving the first $4t$ nodes of $T$ traversed on input $\Pi$, where $\ell_{1}=r$ is the root node and $\ell_{i}$ denotes the node traversed at the $i$th level of $T$. If only $m<4t$ nodes are traversed, define $\ell_{i}=\ell_{m}$ for $i>m$; i.e., $\varphi(\ell_{i})=\varphi(\ell_{m})$. From Lemma 5.16, Markov’s inequality and a union bound, we may now write

[TABLE]

Hence there exists a value $i^{*}$, where $1\leq i^{*}\leq 4t-1$, such that

[TABLE]

But

[TABLE]

where $T_{i^{*}}$ is the set of all nodes at depth $i^{*}$ in $T$, $w_{0},\dots,w_{n}$ are the children of $v$ and ${\textbf{i}}_{v}$ is the random variable giving the score of $F(x_{v})$ on an input $\Pi\in S_{v}$ and [math] otherwise. Since the events $\Pi\in S_{v}$ and $\Pi\in S_{u}$ are disjoint for $v\neq u$, we conclude that there must exist a node $v\in T_{i^{*}}$ for which

[TABLE]

Combined with Lemma 5.7 this shows that $n\log\log n/t=O(1)$; i.e., $t=\Omega(n\log\log n)$. This concludes the proof of Theorem 5.1.

Acknowledgments

Most of this work was done while Benjamin Doerr was with the Max Planck Institute for Informatics (MPII) in Saarbrücken, Germany, and Carola Doerr was with the MPII and Université Diderot, Paris, France.

Carola Doerr acknowledges support from a Feodor Lynen postdoctoral research fellowship of the Alexander von Humboldt Foundation and the Agence Nationale de la Recherche under the project ANR-09-JCJC-0067-01.