Beyond trace reconstruction: Population recovery from the deletion channel

TL;DR

This paper studies the problem of learning a distribution over multiple strings from samples corrupted by a deletion channel, introducing new algorithms and bounds that extend trace reconstruction to population recovery.

Contribution

It introduces the first algorithms and lower bounds for population recovery under deletion noise for multiple strings, extending trace reconstruction techniques.

Findings

Sample complexity is $2^{n^{1/2+o(1)}}$ for small populations.

Lower bounds show $n^{ ext{Omega}( ext{population size})}$ samples are necessary.

New polynomial-based analysis generalizes previous trace reconstruction methods.

Abstract

\emph{Population recovery} is the problem of learning an unknown distribution over an unknown set of $n$-bit strings, given access to independent draws from the distribution that have been independently corrupted according to some noise channel. Recent work has intensively studied such problems both for the bit-flip and erasure noise channels. We initiate the study of population recovery under the \emph{deletion channel}, in which each bit is independently \emph{deleted} with some fixed probability and the surviving bits are concatenated and transmitted. This is a far more challenging noise model than bit-flip~noise or erasure noise; indeed, even the simplest case in which the population size is 1 (corresponding to a trivial distribution supported on a single string) corresponds to the \emph{trace reconstruction} problem, a challenging problem that has received much recent attention (see e.g.~\cite{DOS17,NP17,PZ17,HPP18,HHP18}). We give algorithms and lower bounds for population recovery under the deletion channel when the population size is some $\ell>1$. As our main sample complexity upper bound, we show that for any $\ell=o(\log n/\log \log n)$, a population of $\ell$ strings from $\{0,1\}^n$ can be learned under deletion channel noise using $\smash{2^{n^{1/2+o(1)}}}$ samples. On the lower bound side, we show that $n^{\Omega(\ell)}$ samples are required to perform population recovery under the deletion channel, for all $\ell \leq n^{1/2-\epsilon}$. Our upper bounds are obtained via a robust multivariate generalization of a polynomial-based analysis, due to Krasikov and Roddity \cite{KR97}, of how the $k$-deck of a bit-string uniquely identifies the string; this is a very different approach from recent algorithms for trace reconstruction (the $\ell=1$ case). Our lower bounds build on moment-matching results of Roos~\cite{Roo00} and Daskalakis and Papadimitriou~\cite{DP15}.

Equations

$$\frac{1}{{\epsilon }^2}\cdot {\left(\frac{2}{1-\delta }\right)}^{\sqrt{n}\cdot (\log n{)}^{O(\ell )}}$$

$$E_{\mathbf{z}\leftarrow {\mathrm{Del}}_{\delta }(x)}[{\mathbf{z}}_0],\cdots ,E_{\mathbf{z}\leftarrow {\mathrm{Del}}_{\delta }(x)}[{\mathbf{z}}_{n-1}]$$

$${\mathbf{E}\text{/}}_{\mathbf{j}\in \{1,2\}}{\mathbf{E}\text{/}}_{\mathbf{z}\leftarrow {\mathrm{Del}}_{\delta }(x^{\mathbf{j}})}[{\mathbf{z}}_i]={\mathbf{E}\text{/}}_{\mathbf{j}\in \{1,2\}}{\mathbf{E}\text{/}}_{\mathbf{z}\leftarrow {\mathrm{Del}}_{\delta }(y^{\mathbf{j}})}[{\mathbf{z}}_i],\text{for every i∈{0,…,n−1}.}$$

$$\sum _{0\le i<n}{\delta }_i\cdot p(i)\ne 0.$$

$$\Delta(S)=\operatorname{{\bf Pr}}_{\bm{x}\sim\mathbf{X}}\big{[}\bm{x}_{S}=c\big{]}-\operatorname{{\bf Pr}}_{\bm{y}\sim\mathbf{Y}}\big{[}\bm{y}_{S}=c\big{]}.$$

$$\left|\sum_{0\leq t_{1}<\cdots<t_{d}<n}\phi(t_{1},\ldots,t_{d})\cdot\Delta\big{(}\{t_{1},\ldots,t_{d}\}\big{)}\right|$$

$$\phi(t_{1},\ldots,t_{d})=f\big{(}w_{1}t_{1}+\cdots+w_{d}t_{d}\big{)},$$

$$\sum_{0\leq t_{1}<\cdots<t_{d}<n}\phi(t_{1},\ldots,t_{d})\cdot\Delta\big{(}\{t_{1},\ldots,t_{d}\}\big{)}=\sum_{b=0}^{nd\hskip 1.42271pt\|w\|_{\infty}}f(b)\cdot\Gamma(b),$$

$$d_{\mathrm{TV}}(\mathbf{X},\tilde{\mathbf{X}})={\frac{1}{2}}\sum_{x\in\mathcal{X}}\big{|}\mathbf{X}(x)-\tilde{\mathbf{X}}(x)\big{|},$$

$$\#(v,\mathbf{X})=\mathop{{\bf E}\/}_{\bm{x}\sim\mathbf{X}}\big{[}\#(v,\bm{x})\big{]}.$$

$$\big{(}\mathsf{D}_{k}(x)\big{)}_{v}=\frac{\#(v,x)}{{n\choose k}}.$$

$$\big{(}\mathsf{D}_{k}(\mathbf{X})\big{)}_{v}=\frac{\#(v,\mathbf{X})}{{n\choose k}},$$

$$\frac{1}{{\epsilon }^2}\cdot {\left(\frac{2}{1-\delta }\right)}^{\sqrt{n}\cdot (\log n{)}^{O(\ell )}}$$

$$\mathrm{restrict}(\mathbf{X},T,c):=\operatorname{{\bf Pr}}_{\bm{x}\sim\mathbf{X}}\big{[}(\bm{x}_{t_{1}},\ldots,\bm{x}_{t_{d}})=c\hskip 1.42271pt\big{]},$$

$$\Big{|}\hskip 0.85355pt\mathrm{restrict}(\mathbf{X},T,c)-\mathrm{restrict}(\mathbf{Y},T,c)\hskip 0.85355pt\Big{|}\geq\frac{d_{\mathrm{TV}}(\mathbf{X},\mathbf{Y})}{\ell^{O(\ell)}}.$$

$$\frac{p_{i^{\ast }}}{p_1}=\frac{p_{i^{\ast }}}{p_{i^{\ast }-1}}\cdots \frac{p_2}{p_1}>\frac{1}{(4\ell {)}^{i^{\ast }-1}}$$

$$p_{i^{\ast }}\ge \frac{\epsilon }{(4\ell {)}^{i^{\ast }}}\ge \frac{\epsilon }{(4\ell {)}^{{\ell }^{\prime }}}.$$

$$\Big{|}\hskip 0.85355pt\mathrm{restrict}(\mathbf{X},T,c)-\mathrm{restrict}(\mathbf{Y},T,c)\hskip 0.85355pt\Big{|}\geq p_{i^{*}}-\sum_{i>i^{*}}p_{i}\geq p_{i^{*}}-2\ell\cdot\frac{p_{i^{*}}}{4\ell}=\frac{p_{i^{*}}}{2}\geq\frac{\varepsilon}{\ell^{O(\ell)}}.$$

$$\frac{|\Delta (T_a)|}{|\Delta (T_{a^{\prime }})|}\le \lambda .$$

$${\Delta }_{\mathbf{X},\mathbf{Y},c}(T):=\mathrm{restrict}(\mathbf{X},T,c)-\mathrm{restrict}(\mathbf{Y},T,c).$$

$$(x_{s_1}^1,x_{s_1}^2)=(x_{t_1}^1,x_{t_1}^2)\text{and}(x_{s_2}^1,x_{s_2}^2)=(x_{t_2}^1,x_{t_2}^2).$$

$$(x_i^1,\dots ,x_i^{{\ell }^{\prime }})=u\text{and}(y_i^1,\dots ,y_i^{{\ell }^{\prime \prime }})=v.$$

$$\big{\{}(T_{a},\mathcal{S}_{a}):\text{$\mathcal{S}_{a}$ is nonempty and $\Delta_{\mathbf{X},\mathbf{Y},c}(T_{a})\neq 0$}\big{\}},$$

$$V=\Big{\{}v>0:v=c_{1}u_{1}+\cdots+c_{g}u_{g}\ \hskip 1.42271pt\text{for some $c_{1},\ldots,c_{g}\in\{-1,0,1\}$}\Big{\}}.$$

$$\frac{v_{i+1}}{v_i}\ge (g+2)!,\text{for all i∈[g].}$$

$$v_i=c_{i,1}{u}_1+\cdots +c_{i,g}{u}_g$$

$$v_{i^{\ast }}={\alpha }_1{v}_1+\cdots +{\alpha }_{i^{\ast }-1}{v}_{i^{\ast }-1}\le |{\alpha }_1|\cdot v_1+\cdots +|{\alpha }_{i^{\ast }-1}|\cdot v_{i^{\ast }-1}.$$

$$M=O\left(\frac{k}{{\xi }^2(1-\delta {)}^{2k}}\right)$$

$$\displaystyle\mathop{{\bf E}\/}_{\bm{z}\sim\mathrm{Del}_{\delta}(\mathbf{X})}\big{[}\#(v,\bm{z})\big{]}$$

$$\displaystyle=(1-\delta)^{k}\cdot\mathop{{\bf E}\/}_{\bm{x}\sim\mathbf{X}}\big{[}\#(v,\bm{x})\big{]}=(1-\delta)^{k}\cdot\#(v,\mathbf{X})=(1-\delta)^{k}\cdot{n\choose k}\cdot\big{(}\mathsf{D}_{k}(\mathbf{X})\big{)}_{v}.$$

$$M=O\left(\frac{k}{{\xi }^2(1-\delta {)}^{2k}}\right)$$

Full text

In this paper we initiate the study of population recovery under the deletion channel, in which each bit $b$ is independently deleted with some fixed probability and the surviving bits are concatenated and transmitted. This is a far more challenging noise model than bit-flip noise or erasure noise; indeed, even the simplest case in which the population is of size 1 (corresponding to a trivial probability distribution supported on a single string) corresponds to the trace reconstruction problem, which is a challenging problem that has received much recent attention (see e.g. [DOS17a, NP17, PZ17, HPP18, HHP18]).

In this work we give algorithms and lower bounds for population recovery under the deletion channel when the population size is some value $\ell>1$. As our main sample complexity upper bound, we show that for any population size $\ell=o(\log n/\log\log n)$, a population of $\ell$ strings from $\{0,1\}^{n}$ can be learned under deletion channel noise using $\smash{2^{n^{1/2+o(1)}}}$ samples. On the lower bound side, we show that at least $n^{\Omega{(\ell)}}$ samples are required to perform population recovery under the deletion channel when the population size is $\ell$, for all $\ell\leq n^{1/2-\varepsilon}$.

Our upper bounds are obtained via a robust multivariate generalization of a polynomial-based analysis, due to Krasikov and Roddity [KR97], of how the $k$-deck of a bit-string uniquely identifies the string; this is a very different approach from recent algorithms for trace reconstruction (the $\ell=1$ case). Our lower bounds build on moment-matching results of Roos [Roo00] and Daskalakis and Papadimitriou [DP15].

1 Introduction

In recent years the unsupervised learning problem of population recovery has emerged as a significant focus of research attention in theoretical computer science [DRWY12, MS13, BIMP13, LZ15, DST16, WY16, PSW17, DOS17b]. In the population recovery problem there is an unknown distribution $\mathbf{X}$ over an unknown set of $n$-bit strings from $\{0,1\}^{n}$, and the learner’s job is to reconstruct a high-accuracy approximation of $\mathbf{X}$ given access to noisy independent draws from $\mathbf{X}$ (so each data point which the learning algorithm receives is independently generated as follows: an $n$-bit string is drawn from $\mathbf{X}$ and corrupted by some noise process, and the result is provided to the learning algorithm). The two noise models which have chiefly been studied to date are the bit-flip noise model, in which each coordinate is independently flipped with some fixed probability, and the erasure noise model, in which each coordinate is independently replaced by ‘?’ with some fixed probability.

Since the population recovery problem was first introduced in [DRWY12, WY16], a number of positive results and lower bounds have been obtained for different variants of the problem. In one popular version of the problem [PSW17, DOS17b, MS13], for a particular noise model (bit-flip or erasure) the distribution $\mathbf{X}$ may be an arbitrary distribution over $\{0,1\}^{n}$, and the goal is to learn the distribution $\mathbf{X}$ with respect to $\ell_{\infty}$ distance (i.e. to output a list of strings $x^{1},\dots,x^{r}\in\{0,1\}^{n}$ and associated weights $\tilde{\mathbf{X}}(x^{i})$ such that $|\mathbf{X}(x^{i})-\tilde{\mathbf{X}}(x^{i})|\leq\varepsilon$ for all $i\in[r]$ and $\mathbf{X}(x)\leq\varepsilon$ for all $x\in\{0,1\}^{n}\setminus\{x^{1},\dots,x^{r}\}$). In another well-studied version of the problem [WY16, LZ15, DST16], which is closely related to the problems we shall consider, the distribution $\mathbf{X}$ is promised to be supported on at most $\ell$ strings in $\{0,1\}^{n}$ (i.e. the “population size” is promised to be at most $\ell$), and the goal is to output a hypothesis distribution $\tilde{\mathbf{X}}$ over $\{0,1\}^{n}$ which has total variation distance at most $\varepsilon$ from $\mathbf{X}$. Significant progress has been made on determining the sample complexity of population recovery for both of these variants under the bit-flip and erasure noise models; we refer the interested reader to [DST16, PSW17, DOS17b] for the current state of the art.

This work: Population recovery from the deletion channel and its relation to trace reconstruction. In both the bit-flip noise model and the erasure noise model, all of the challenge in the population recovery problem stems from the fact that given a noisy draw from $\mathbf{X}$ it is a priori not clear which element of $\mathbf{X}$’s support was corrupted by noise to produce the noisy draw. Putting it another way, if the population size is promised to be $\ell=1$, then under either of these two noise models it is trivially easy to learn a single unknown string from noisy examples.

In this work we study population recovery under the deletion noise model, which is far more challenging to handle than either bit-flip noise or erasure noise. The deletion channel is defined as follows: when a string $x$ is passed through the deletion channel with deletion parameter $\delta$, each coordinate $x_{i}$ is independently deleted with probability $\delta$, the surviving coordinates are concatenated, and the resulting string (of length $\bm{n}^{\prime}\leq n$, where $\bm{n}^{\prime}$ is distributed as $\mathrm{Bin}(n,1-\delta)$) is the output of the noise process. Intuitively, the deletion channel is challenging because given a received word obtained by passing $x$ through the $\delta$-deletion channel (often referred to as a trace of $x$, and denoted by $\bm{z}\leftarrow\mathrm{Del}_{\delta}(x)$), it is not clear which coordinate of $x$ gave rise to which coordinate of $\bm{z}$. Indeed, in contrast with the bit-flip and erasure noise models, even if the population size is guaranteed to be $\ell=1$, the problem of recovering a single unknown string from independent traces is a well-known and challenging open problem, known as the trace reconstruction problem [Lev01b, Lev01a, BKKM04, KM05, HMPW08, VS08, MPV14, DOS17a, NP17, PZ17, HPP18, HHP18].

There are several motivations for the study of population recovery under the deletion noise model. One motivation is the considerable recent research interest both in the trace reconstruction problem (the $\ell=1$ case of population recovery under the deletion channel) and in population recovery problems under bit-flip and erasure models. Further motivation comes from potential relevance of the deletion channel population recovery problem both to recovery problems in computational biology and to other topics such as DNA data storage. Regarding biological recovery problems, considering population recovery (the $\ell>1$ case) rather than trace reconstruction (the $\ell=1$ case) relaxes the potentially unrealistic assumption that all of the received samples (of a protein sequence, DNA sequence, etc.) are derived from a single unknown target sequence rather than from multiple unknown sequences. Heuristic algorithms for population recovery-type problems have also been applied to DNA storage [OAC*+*18]. In these settings, each string in the population comes from a DNA sequence and the noisy channel can inflict a variety of errors including bit-flips and deletions.

Thus, the authors feel that the time is ripe for a theoretical study of population recovery under the challenging deletion model. In this paper we initiate such a study, obtaining sample complexity upper and lower bounds when the population is of size $\ell>1.$ Before describing our results for populations of size $\ell$ (equivalently, target distributions supported on at most $\ell$ strings), we first recall known upper and lower bounds for the trace reconstruction problem ($\ell=1$) below.

Known bounds on trace reconstruction. The trace reconstruction problem was raised more than fifteen years ago [Lev01b, Lev01a, BKKM04], though in fact some variants of the problem go back at least to the 1970s [Kal73]. The first algorithm that provably succeeds with high probability in reconstructing an arbitrary $\smash{x\in\{0,1\}^{n}}$ using subexponentially many traces is due to Mitzenmacher et al. [HMPW08], who showed that $\smash{2^{\tilde{O}(\sqrt{n})}}$ many traces suffice for any constant deletion rate $\delta$ bounded away from $1$. This result was improved in recent simultaneous and independent works of De et al. [DOS17a] and Nazarov and Peres [NP17]; these papers each showed that for any constant $\delta$ bounded away from $1$, at most $\smash{2^{O(n^{1/3})}}$ traces suffice to reconstruct any $x\in\{0,1\}^{n}$.111Hartung, Holden and Peres [HHP18] have recently extended this result to certain more general regimes where there can be different deletion probabilities for different coordinates and symbols.

Due to the seeming difficulty of the worst-case trace reconstruction problem (reconstructing an arbitrary $x\in\{0,1\}^{n}$), an average-case version of the problem (reconstructing a randomly chosen string $x\in\{0,1\}^{n}$), which turns out to be significantly easier in terms of sample complexity, has also received considerable attention. A number of early works [BKKM04, KM05, VS08] gave efficient algorithms that succeed for trace reconstruction of almost all $x\in\{0,1\}^{n}$ when the deletion rate $\delta$ is sufficiently low ($o_{n}(1)$ as a function of $n$). In [HMPW08] Mitzenmacher et al. gave an algorithm which uses $\mathrm{poly}(n)$ traces to perform average-case trace reconstruction when the deletion rate $\delta$ is at most some sufficiently small constant. Recently the best results on average-case trace reconstruction have been significantly strengthened in works of Peres and Zhai [PZ17] and Holden, Pemantle and Peres [HPP18] which build on the worst-case trace reconstruction results of [DOS17a, NP17]. The latter of these papers [HPP18] gives an algorithm which uses $\smash{\exp((\log n)^{1/3})}$ traces to reconstruct a random $x\in\{0,1\}^{n}$ when the deletion rate is any constant bounded away from 1.

In terms of lower bounds, it is easy to see that if the deletion rate $\delta$ is at least some positive constant, then until $\Omega(\log n)$ draws have been received there will be some bits of the target string $x$ about which no information has been received. Improving on this simple $\Omega(\log n)$ lower bound, McGregor et al. [MPV14] established a sample complexity lower bound of $\Omega(n)$ traces for any constant deletion rate. This was recently improved by Holden and Lyons [HL18] to $\tilde{\Omega}(n^{5/4})$.

Summarizing, for any constant deletion probability $0<\delta<1$ there is currently an exponential gap between the best lower bound of $\smash{\tilde{\Omega}(n^{5/4})}$ samples and the best upper bound of $\smash{2^{O(n^{1/3})}}$ samples for trace reconstruction of an arbitrary string $x\in\{0,1\}^{n}.$

1.1 Our results

Positive result. As our main positive result, we obtain an algorithm which learns any unknown distribution $\mathbf{X}$ supported on at most $\ell$ strings under the deletion channel. For any constant $\ell$ (and in fact even for $\ell$ as large as $o(\log n/\log\log n)$, its sample complexity is exponential in $\smash{{n^{1/2+o(1)}}}$. In more detail, our main positive result is the following:

Theorem 1** (Learning an arbitrary mixture of $\ell$ strings under the deletion channel).**

There is an algorithm with the following performance guarantee: Let $\mathbf{X}$ be an arbitrary distribution over at most $\ell$ strings in $\{0,1\}^{n}$. For any deletion rate $0<\delta<1$ and any accuracy parameter $\varepsilon$, if the algorithm is given access to independent draws from $\mathbf{X}$ that are independently corrupted with deletion noise at rate $\delta$, then the algorithm uses

[TABLE]

many samples and with probability at least $0.99$ outputs a hypothesis $\tilde{\mathbf{X}}$ which is supported over at most $\ell$ strings and has total variation distance at most $\varepsilon$ from the unknown target distribution $\mathbf{X}$.

It is easy to see that if the target distribution is promised to be uniform over (a multi-set of) at most $\ell$ strings, then the algorithm of Theorem 1 can be used to exactly reconstruct the unknown multi-set. As we explain in Section 2, while Theorem 1 extends prior results on trace reconstruction (the $\ell=1$ case), it is proved using very different techniques from recent works [HMPW08, DOS17a, NP17, PZ17, HPP18, HHP18] on trace reconstruction.

We note that for deletion rates $\delta$ that are bounded away from 1 by a constant, the $\smash{2^{O(n^{1/3})}}$ sample complexity bounds of [DOS17a, NP17] for trace reconstruction are better than the $\ell=1$ case of our result. However, our bounds apply even if the deletion rate $\delta$ is very close to 1; in particular, [DOS17a, NP17] give no results for very high deletion rates $\delta=1-o(1/\sqrt{n})$, while Theorem 1 gives a $\smash{2^{\tilde{O}(\sqrt{n})}}$ bound for $\delta=1-1/2^{\mathrm{polylog}(n)}$ and a $2^{o(n)}$ bound even for $\delta$ as large as $\smash{1-1/2^{\sqrt{n}/\mathrm{polylog}(n)}}.$ Of course, the main feature of Theorem 1 is that it applies when $\ell>1$ (unlike [DOS17a, NP17]).

Negative result. Complementing the sample complexity upper bound, we obtain a lower bound on the sample complexity of population recovery. Our lower bound shows that for a wide range of values of $\ell$, at least $n^{\Omega(\ell)}$ samples are required when the population is of size at most $\ell$. An informal version of our lower bound is as follows (see Theorem 6 in Section 5 for a detailed statement):

Theorem 2** (Sample complexity lower bound, informal statement).**

Let $0<\delta<1$ be any constant deletion probability and suppose that $A$ is an algorithm which, when run on samples drawn from the $\delta$-deletion channel over an arbitrary distribution $\mathbf{X}$ supported over at most $\ell\leq n^{0.499}$ many strings, with probability at least $0.51$ outputs a hypothesis distribution $\tilde{\mathbf{X}}$ that has total variation distance at most $0.49$ from the unknown target distribution $\mathbf{X}$. Then $A$ must use $n^{\Omega(\ell)}$ many samples.

2 Our techniques

As noted earlier, our positive result (Theorem 1) gives a sample complexity upper bound for the original $(\ell=1)$ trace reconstruction problem as a special case. We remark that both of the recent $\smash{2^{O(n^{1/3})}}$ sample complexity upper bounds for the trace reconstruction problem [DOS17a, NP17], as well as the earlier work of [HMPW08], employed essentially the same algorithmic approach, which is referred to in [DOS17a] as a “mean-based algorithm.” At a high level, mean-based algorithms use their samples (traces) only to compute empirical estimates of the $n$ expectations222In this context, the original unknown target string $x$ is viewed as belonging to $\{-1,1\}^{n}$, and a trace $z$ obtained from $\mathrm{Del}_{\delta}(x)$ is viewed as a string in $\smash{\{-1,1\}^{n^{\prime}}}$ for some $n^{\prime}\leq n$ with $n-n^{\prime}$ zeros appended to the end. Throughout the paper, we use $[0:n-1]=\{0,\ldots,n-1\}$ to index entries of a string of length $n$.

[TABLE]

corresponding to the coordinate means of the received traces; they then only use those $n$ estimates to reconstruct the unknown target string $x$. Both of the algorithms in [DOS17a, NP17], as well as the algorithm from [HMPW08] for trace reconstruction from an arbitrary string $x$, are mean-based algorithms. (Both [DOS17a] and [NP17] show that their sample complexity upper bounds are essentially best possible for any mean-based trace reconstruction algorithm.)

While mean-based algorithms have led to state-of-the-art results for trace reconstruction of a single string, this approach breaks down even for the simplest non-trivial cases of population recovery under the deletion channel. Indeed, even when $\ell=2$ and the unknown distribution $\mathbf{X}$ is promised to be uniform over two strings, it is easy to see that the coordinate means do not provide enough information to recover $\mathbf{X}$. For example, if $(x^{1},x^{2})$ and $(y^{1},y^{2})$ are two pairs of strings whose sums (as vectors in $\mathbbm{R}^{n}$) $x^{1}+x^{2}$ and $y^{1}+y^{2}$ are equal (such as $x^{1}=0^{n},$ $x^{2}=1^{n}$, $y^{1}=0^{n/2}1^{n/2},$ $y^{2}=1^{n/2}0^{n/2}$), it is easy to see that the coordinate means of received traces will match perfectly:

[TABLE]

Thus the mean-based approach of [HMPW08, DOS17a, NP17] does not suffice for even the simplest version of the population recovery problem when $\ell=2$. Indeed, our sample complexity upper bounds are obtained using a completely different approach, which we explain below.

2.1 Warm-up: A different approach to trace reconstruction (the $\ell=1$ case)

As a warm-up to our main results, we first give a high-level explanation of how our approach can be used to obtain a simple $\smash{2^{\tilde{O}(\sqrt{n})}}$-sample algorithm for the trace reconstruction problem. While this is a higher sample complexity than the state-of-the-art mean-based approach of [DOS17a, NP17] (though our approach does better for very high deletion rates, as noted earlier), our approach has the crucial advantage that it can be adapted to go beyond the $\ell=1$ case, whereas the mean-based approach cannot handle $\ell>1$ as described above.

In a nutshell, the essence of our approach is to work with subsequence frequencies in the original string $x$ (in contrast, note that the mean-based approach uses single-coordinate frequencies in the received traces). To explain further we introduce some useful terminology: the $k$-deck of a string $x\in$ $\{0,1\}^{n}$, denoted $\mathsf{D}_{k}(x)$, is the multi-set of all ${n\choose k}$ subsequences of $x$ with length exactly $k$. Thus, the $k$-deck encapsulates all frequency information about length-$k$ subsequences of $x$.

A question that arises naturally in the combinatorics of words is the following: what is the smallest value of $k$ (as a function of $n$) so that for every string $x\in\{0,1\}^{n}$, the $k$-deck of $x$ uniquely identifies $x$? Despite significant investigation dating back to the 1970s [Kal73], this basic quantity is still poorly understood. Improving on earlier $k\leq n/2$ bounds of Kalashnik [Kal73] and Manvel et al. [MMS*+*91] and a simultaneous $k=O(\sqrt{n\log n})$ bound of Scott [Sco97], Krasikov and Roddity [KR97] showed that $k=O(\sqrt{n})$ suffices. On the lower bounds side, the best lower bound known is $\smash{k=2^{\Omega(\sqrt{\log n})}}$, due to Dudík and Schulman [DS03] (improving on earlier $k=\Omega(\log n)$ lower bounds of [MMS*+*91] and [CK97]).

The relevance of upper bounds on $k$ to the trace reconstruction problem is intuitively clear, and indeed, McGregor et al. [MPV14] observed that if the deletion rate $\delta$ is at most $1-c\sqrt{({\log n})/n}$, then it is trivially easy to extract a random length-$O(\sqrt{n\log n})$ subsequence of $x$ from a typical trace of $x$. Combining this with the $k=O(\sqrt{n\log n})$ upper bound of Scott [Sco97] and a straightforward sampling-based procedure (which estimates the frequency of each string in $\{0,1\}^{k}$ to high enough accuracy to determine its exact multiplicity in the $k$-deck), they obtained an information-theoretic sample complexity upper bound on trace reconstruction: for $\delta\leq 1-c\sqrt{({\log n})/n}$, at most $\smash{n^{O(\sqrt{n\log n})}}$ traces suffice to reconstruct any $x\in\{0,1\}^{n}$ with high probability.

As an initial observation, we slightly strengthen the [MPV14] result by showing that for any value of $\delta<1$, an algorithm which combines sampling and dynamic programming can exactly infer the $k$-deck of an unknown string $x\in\{0,1\}^{n}$ with high probability using $\left({n/({1-\delta})}\right)^{O(k)}$ traces from $\mathrm{Del}_{\delta}(x).$ (See Theorem 4 for a detailed statement and proof of a more general version of this result.) Combining this with the [KR97] upper bound $k=O(\sqrt{n})$, we get that any string $x$ can be reconstructed from $\delta$-deletion noise using $\smash{\left(n/({1-\delta})\right)^{O(\sqrt{n})}}$ samples.

The above-outlined approach to trace reconstruction (the $\ell=1$ case of population recovery) is the starting point for our main positive result, Theorem 1. In the next subsection we give a high-level description of some of the challenges that arise in dealing with multiple strings and how this work overcomes them.

2.2 Ingredients in the proof of Theorem 1

Recall that in the setting of Theorem 1 the unknown $\mathbf{X}$ is an arbitrary distribution supported on at most $\ell$ strings $x^{1},\dots,x^{\ell}$ in $\{0,1\}^{n}$. Viewing $\mathbf{X}$ as a mixture of individual strings, there is a natural notion of the $k$-deck of $\mathbf{X}$, which we denote by $\mathsf{D}_{k}(\mathbf{X})$ and which is the weighted multi-set corresponding to the $\mathbf{X}$-mixture of the decks $\mathsf{D}_{k}(x^{1}),\dots,\mathsf{D}_{k}(x^{\ell}).$333By a weighted-multiset we mean a multiset in which each element has a weight. Alternatively, one can interpret (after normalization) $\mathsf{D}_{k}(x)$ as a probability distribution over the $2^{k}$ strings in $\{0,1\}^{k}$ and in this case, $\mathsf{D}_{k}(\mathbf{X})$ can be viewed as a probability distribution that is the $\mathbf{X}$-mixture of $\mathsf{D}_{k}(x^{1}),\dots,\mathsf{D}_{k}(x^{\ell})$.As a result, Theorem 1 will follow if we can show the following: if two distributions $\mathbf{X},\mathbf{Y}$ over $\{0,1\}^{n}$ (each supported on at most $\ell$ strings) have $d_{\mathrm{TV}}(\mathbf{X},\mathbf{Y})>\varepsilon$, then for a not-too-large value of $k$, the $k$-decks $\mathsf{D}_{k}(\mathbf{X})$ and $\mathsf{D}_{k}(\mathbf{Y})$ (note that these are two weighted multi-sets of strings in $\{0,1\}^{k}$) must be “noticeably different.” This is established in Lemma 4.6, which is the technical heart of our upper bound.

To explain our proof of Lemma 4.6 it is useful to revisit the $\ell=1$ setting; the analogous (and much easier to prove) statement in this context is that given any two strings $x\neq y\in\{0,1\}^{n}$, the $k$-decks $\mathsf{D}_{k}(x)$ and $\mathsf{D}_{k}(y)$ are not identical when ${k\geq C\sqrt{n}}$ for some large enough constant $C$. This is the main result of [KR97] (and a similar statement, with a slightly weaker quantitative bound on $k$, is also proved in [Sco97]). Since the $k$-deck in and of itself is somewhat difficult to work with (being a multi-set over $\{0,1\}^{k}$), both [KR97] and [Sco97] work instead with the summed $k$-deck, which we denote by $\mathsf{SD}_{k}(x)$ and which is simply the vector in $\mathds{N}^{k}$ obtained by summing all ${n\choose k}$ elements of the $k$-deck $\mathsf{D}_{k}(x)$ (recall that each element of $\mathsf{D}_{k}(x)$ is a vector in $\{0,1\}^{k}$). Both [KR97] and [Sco97] actually show that for a suitable value of $k$, the summed $k$-deck $\mathsf{SD}_{k}(x)$ uniquely identifies $x$ among all strings in $\{0,1\}^{n}$. (Both papers also observe that by a simple counting argument, the smallest such $k$ is at least $\tilde{\Omega}(\sqrt{n}).$) The [KR97] proof reduces the analysis of the summed $k$-deck to an extremal problem about univariate polynomials. The key ingredient of their proof is the following result about univariate polynomials, which was established in [BEK99] in their work on the Prouhet-Tarry-Escott problem:

Given any nonzero vector $\delta\in\{-1,0,1\}^{n}$, there is a univariate polynomial $p$ of degree $O(\sqrt{n})$ such that

[TABLE]

Setting $\delta=x-y\neq 0$, to finish the proof of $\mathsf{SD}_{k}(x)\neq\mathsf{SD}_{k}(y)$ when $x\neq y$ and $k\geq C\sqrt{n}$, [KR97] shows that choosing $k$ to be $\deg(p)+1$, the inequality ($\dagger$ ‣ 2.2) implies that $\mathsf{SD}_{k}(x)\neq\mathsf{SD}_{k}(y)$.

Returning to our $\ell$-string setting, we remark that several challenges arise which are not present in the one-string setting. To highlight one of these, due to the difficulty of analyzing the entire $k$-deck of $\mathbf{X}$ it is natural to try to work with the summed $k$-deck $\mathsf{SD}_{k}(\mathbf{X})$ (a nonnegative vector in $\mathbbm{R}^{k}$), which is obtained by summing all elements of the weighted multi-set $\mathsf{D}_{k}(\mathbf{X})$. Indeed it can be shown via a rather straightforward extension of the [KR97] analysis that, when $\mathbf{X}$ is uniform over $x^{1},\ldots,x^{\ell}$, the summed $k$-deck with $k=O(\sqrt{n}\log\ell)$ suffices to exactly reconstruct the sum $x^{1}+\cdots+x^{\ell}$ (a vector in $\mathds{N}^{n}$). But even for uniform distributions, a difficulty which arises is that the summed $k$-deck (even with $k=n$) cannot distinguish between two uniform distributions over $x^{1},\dots,x^{\ell}$ versus $y^{1},\dots,y^{\ell}$ that have the same coordinate-wise sums, i.e. that satisfy $x^{1}+\cdots+x^{\ell}=y^{1}+\cdots+y^{\ell}$.444This is conceptually similar to the inability of mean-based algorithms to handle multiple strings noted earlier. Indeed, considering the same example as earlier, in which $\ell=2$ and $x^{1}=0^{n},\ x^{2}=1^{n},y^{1}=0^{n/2}1^{n/2}$ and $y^{2}=1^{n/2}0^{n/2}$, the summed $k$-deck is $({n\choose k},\dots,{n\choose k})/2\in\mathbbm{R}^{k}$ in both cases.

At a high level our Lemma 4.6 can be viewed as a robust generalization of the [KR97] result. A key technical ingredient in its proof is a robust generalization of the [BEK99] result to multivariate polynomials. (The summed $k$-deck corresponds to univariate polynomials, so at a high level our analysis involving multivariate polynomials can be viewed as how we get around the obstacle noted in the previous paragraph.) The proof of Lemma 4.6 consists of three steps which we outline below.

The first conceptual step of our argument is to show that if two support-$\ell$ distributions $\mathbf{X}$ and $\mathbf{Y}$ over $\{0,1\}^{n}$ satisfy $d_{\mathrm{TV}}(\mathbf{X},\mathbf{Y})\geq\varepsilon$, then there exists a subset $T\subset[0:n-1]$ of size $d=\lfloor\log(2\ell)\rfloor$ such that $\mathbf{X}$ and $\mathbf{Y}$ “differ significantly” just on the coordinates in $T$. In particular, there is some $|T|$-bit string $c$ such that $\operatorname{{\bf Pr}}_{\bm{x}\sim\mathbf{X}}[\hskip 0.85355pt\bm{x}_{T}=c\hskip 0.85355pt]$ is significantly different from $\operatorname{{\bf Pr}}_{\bm{y}\sim\mathbf{Y}}[\hskip 0.85355pt\bm{y}_{T}=c\hskip 0.85355pt]$, where we use $x_{T}$ to denote the restriction of a string $x\in\{0,1\}^{n}$ on coordinates in $T$. (This is made precise in Lemma 4.1.) Let $\smash{\Delta:{[0:n-1]\choose d}\rightarrow\mathbbm{R}}$ be the following function over size-$d$ subsets of $[0:n-1]$:

[TABLE]

Then Lemma 4.1 implies that $\|\Delta\|_{\infty}$ is not too small.

The second (and central) conceptual step of our argument can be viewed as a robust generalization of the [BEK99] result to $d$-variate polynomials, as alluded to earlier. The key result giving this step, Lemma 4.7, roughly speaking states the following:

Given the $\Delta$ as defined in (2), there is a $d$-variate polynomial $\phi$ of not-too-high degree (roughly $\sqrt{n}$) such that555The reader who has peeked ahead to the statement of Lemma 4.7 may have noticed that the lemma statement also bounds the magnitudes of coefficients of the polynomial $\phi$. This is done for technical reasons, and we skip these technical details in the high-level description here.

[TABLE]

can be lower bounded in terms of $\|\Delta\|_{\infty}$, which is not too small by Lemma 4.1.

The third conceptual step relates ($\dagger\dagger$ ‣ 2.2) to the distance between the $k$-decks $\mathsf{D}_{k}(\mathbf{X})$ and $\mathsf{D}_{k}(\mathbf{Y})$, by showing that if ($\dagger\dagger$ ‣ 2.2) is not too small then $\mathsf{D}_{k}(\mathbf{X})$ and $\mathsf{D}_{k}(\mathbf{Y})$ must be “noticeably different” when $k$ is chosen to be $\deg(\phi)+d$. We refer the reader to Lemma 4.8. At a high level this is analogous to, but technically more involved than, the [KR97] proof that the inequality ($\dagger$ ‣ 2.2) for $\delta=x-y$ implies that $\mathsf{SD}_{k}(x)\neq\mathsf{SD}_{k}(y)$ with $k=\deg(p)+1$. Lemma 4.6 then follows by combining all three steps, i.e. $d_{\mathrm{TV}}(\mathbf{X},\mathbf{Y})$ being large implies that $\mathsf{D}_{k}(\mathbf{X})$ is “noticeably different” from $\mathsf{D}_{k}(\mathbf{Y})$ for $k$ that is roughly $\sqrt{n}$. Below we outline the main ingredients needed in the second step.

In the search for a low-degree polynomial $\phi$ such that the sum in ($\dagger\dagger$ ‣ 2.2) has large magnitude, it is natural to define $\phi(t_{1},\ldots,t_{d})$ by first projecting $(t_{1},\ldots,t_{d})$ to a line and then applying a univariate polynomial similar to the $p$ used in ($\dagger$ ‣ 2.2). To make this more precise, we will look for $\phi$ of the form

[TABLE]

where $w_{1},\ldots,w_{d}$ are positive integers (so the line is along the direction $w=(w_{1},\ldots,w_{d})$) and $f$ is a low-degree univariate polynomial to be specified later. With (3), we rewrite the sum in ($\dagger\dagger$ ‣ 2.2) as

[TABLE]

where $\Gamma(b)$ is the sum of $\Delta(T)$ over all $d$-subsets $T=\{t_{1},\ldots,t_{d}\}$ such that $0\leq t_{1}<\cdots<t_{d}<n$ and $b=w_{1}t_{1}+\cdots$ $+w_{d}t_{d}$. Comparing (4) with ($\dagger$ ‣ 2.2), our goal would follow directly from the [BEK99] result by choosing $f$ to be $p$ if $\Gamma$ is nonzero and takes values in $\{-1,0,1\}$ (or even $\{-c,0,c\}$ for some not too small $c>0$). However, the main difficulty we encounter is that $\Gamma$ is much more complex than the $\{-1,0,1\}^{n}$ vectors that can be handled by techniques of [BEK99]; for example, $\Gamma$ in general may contain a large number (depending on $n$) of distinct values.

There are three ingredients we use in choosing $w_{1},\ldots,w_{d}$ and $f$ to overcome this difficulty:

1. (A)

We first observe that $\Delta$ has a combinatorial “rectangular” structure, which implies that the support of $\Delta$ can be partitioned into a small number of sets $\mathcal{S}_{1},\mathcal{S}_{2},\ldots$ (each element of $\mathcal{S}_{a}$ is a size-$d$ subset of $[0:n-1]$) such that all $T\in\mathcal{S}_{a}$ share the same value of $\Delta(T)$ and there is a set $T_{a}\in\mathcal{S}_{a}$ that is dominated666Given two size-$d$ subsets $S=\{s_{1},\ldots,s_{d}\}$ and $T=\{t_{1},\ldots,t_{d}\}$ of $[0:n-1]$ with $s_{1}<\cdots<s_{d}$ and $t_{1}<\cdots<t_{d}$, we say that $S$ is dominated by $T$ if $s_{i}\leq t_{i}$ for all $i$. by every $T\in\mathcal{S}_{a}$. We refer to $T_{a}$ as the anchor set of $\mathcal{S}_{a}$. This is made precise in Lemma 4.3. Moreover, we show in Lemma 4.4 that the collections $\mathcal{S}_{a}$ can be divided into an even smaller number of groups such that, for any $\mathcal{S}_{a},\mathcal{S}_{a^{\prime}}$ that belong to the same group, the ratio of $|\Delta(T_{a})|$ and $|\Delta(T_{a^{\prime}})|$ is bounded from above by a small number. 2. (B)

Next we observe that when $w_{1},\ldots,w_{d}$ are drawn from a suitable distribution, all anchor sets in (A) have distinct images after the projection. (See Claim 4.9.) We fix such a tuple $(w_{1},\ldots,w_{d})$. (A) and (B) together are then used to obtain (see Lemma 4.11) a strong structural characterization of $\Gamma$. 3. (C)

Finally we define a new univariate polynomial $f$ based on Chebyshev polynomials and the construction of $p$ in [BEK99]. (See Lemma 4.10.) The characterization of $\Gamma$ and properties of $f$ are then combined to finish the proof by showing that the sum in ($\dagger\dagger$ ‣ 2.2) has not too small magnitude when we apply the polynomial $\phi$ given in (3).

2.3 Our lower bounds

We begin by recalling the $\Omega(n)$ lower bound of McGregor et al. [MPV14]. This lower bound is obtained via a simple analysis of the two distributions of traces resulting from the two strings $x^{1}=0^{n/2}10^{n/2-1}$ and $x^{2}=0^{n/2-1}10^{n/2}$. The starting point of the [MPV14] analysis is the observation that under the $\delta$-deletion channel, conditioned on the sole “1” coordinate being retained, the distribution of a trace of $x^{1}$ corresponds to $(\bm{a},\bm{b})$ where $\bm{a}$ and $\bm{b}$ are independent draws from $\mathrm{Bin}(n/2,1-\delta)$ and $\mathrm{Bin}(n/2-1,1-\delta)$ respectively, whereas the distribution of a trace of $x^{2}$ corresponds to $(\bm{b},\bm{a}).$ [MPV14] used this to show that the squared Hellinger distance between these two distributions of traces is $O(1/n)$, and in turn use this squared Hellinger distance bound to infer an $\Omega(n)$ sample complexity lower bound for determining whether a collection of received traces came from $x^{1}$ or from $x^{2}$.

Our lower bound approach may be viewed as an extension of the [MPV14] lower bound to mixtures of distributions similar to the ones they consider. The high-level idea of our lower bound proof is as follows: we show that there exist two distributions $\mathbf{X},\mathbf{Y}$ over $\{0,1\}^{n}$ (in fact, over $n$-bit strings with precisely one 1) which have disjoint supports, each of size at most ${2\ell}$, but are such that the total variation distance $d_{\mathrm{TV}}(\mathrm{Del}_{\delta}(\mathbf{X}),\mathrm{Del}_{\delta}(\mathbf{Y}))$, between traces of strings drawn from $\mathbf{X}$ versus traces of strings drawn from $\mathbf{Y}$, is very small. This is easily seen to imply Theorem 2.

For simplicity in introducing the main ideas of our analysis, in this expository overview we will first consider an “$n=+\infty$” version of our population recovery scenario. We begin by considering the distribution $\mathrm{Del}_{\delta}(\tilde{e}_{m+i})$ where $m$ is some fixed value and $\tilde{e}_{m+i}$ is an infinite string with a single $1$ in position $m+i$ and all other coordinates [math]. A $\delta$ fraction of the outcomes of $\mathrm{Del}_{\delta}(\tilde{e}_{m+i})$ are the infinite all-[math] string, which conveys no information. The other $1-\delta$ fraction of the outcomes each have precisely one 1, occurring in position $1+\bm{a}$ where $\bm{a}$ is distributed according to the binomial distribution $\mathrm{Bin}(m+i,1-\delta)$. In this infinite-$n$ setting, two distributions $\mathbf{X},\mathbf{Y}$ over strings of the form $\tilde{e}_{m+i}$ with disjoint supports correspond to two mixtures of distinct binomial distributions (all with second parameter $1-\delta$, but with a set of first parameters in the first mixture that is disjoint from the set of first parameters in the second mixture). The animating idea behind our construction and analysis is that it is possible for two distinct mixtures of binomials like this to be very close to each other in total variation distance.777We remark that our actual scenario is more complicated than this idealized version because $n$ is a finite value rather than $+\infty$. For $n=2m+1$, this means that a received trace $0^{\bm{a}}10^{\bm{b}}$ which contains a 1 and came from $\mathrm{Del}_{\delta}(e_{m+i})$ provides a pair of values $(\bm{a},\bm{b})$ where $\bm{a}$ is distributed according to $\mathrm{Bin}(m+i,{\rho})$ and $\bm{b}$ is independently distributed according to $\mathrm{Bin}(m-i,{\rho})$ where $\rho=1-\delta$ is the retention probability. This second value $\bm{b}$ provides additional information which is not present in the $n=+\infty$ version of the problem, and this makes it more challenging and more technically involved to prove a lower bound. We deal with these issues in Section 5.2.

In order to show that two distinct mixtures of binomial distributions as described above can be very close to each other in total variation distance, our lower bounds employ technical machinery due to Roos [Roo00] and Daskalakis and Papadimitriou [DP15]. Roos [Roo00] developed a “Krawtchouk expansion” which provides an exact expression for the probability that a Poisson binomial distribution (a sum of $n$ independent Bernoulli random variables with expectations $p_{1},\dots,p_{n}$) puts on any given outcome in $\{0,1,\cdots,n\}$. Daskalakis and Papadimitriou [DP15] used Roos’s Krawtchouk expansion to show that under mild technical conditions, low-order moments of any Poisson binomial distribution essentially determine the entire distribution. In more detail, their main result is that if $\mathbf{X},\mathbf{Y}$ are two Poisson binomial distributions (satisfying mild technical conditions) whose $t$-th moments match, i.e. $\operatorname{{\bf E}}[\mathbf{X}^{t}]=\operatorname{{\bf E}}[\mathbf{Y}^{t}]$ for $t=1,\dots,O(\log(1/\varepsilon))$, then the total variation distance between $\mathbf{X}$ and $\mathbf{Y}$ is at most $\varepsilon.$

Our analysis proceeds in two main steps. In the first step, we show that there exist two mixtures of pairs of binomial distributions, which we denote by $\mathbf{D}_{S}$ and $\mathbf{D}_{T}$, with certain desirable properties. $S$ and $T$ are both subsets of $\{0,\dots,2\ell\}$, and $\mathbf{D}_{S}$ is a certain mixture of pairs of binomial distributions $(\mathrm{Bin}(n/2+i,1-\delta),\mathrm{Bin}(n/2-i,1-\delta))$ for $i\in S$ while $\mathbf{D}_{T}$ is a certain mixture of pairs of binomial distributions $(\mathrm{Bin}(n/2+j,1-\delta),\mathrm{Bin}(n/2-j,1-\delta))$ for $j\in T$. We establish the existence of disjoint sets $S,T$ such that the resulting mixtures $\mathbf{D}_{S}$ and $\mathbf{D}_{T}$ have matching $t$-th moments for all $t=1,\dots,\ell.$ This is proved using known algebraic expressions for the moments of binomial distributions and simple linear algebraic arguments. In the second main step, we extend the analysis of Daskalakis and Papadimitriou [DP15] and apply this extension to our setting, in which we are dealing with mixtures of (pairs of) binomial distributions (as opposed to their and Roos’s setting of Poisson binomial distributions). We show that the matching first $\ell$ moments of $\mathbf{D}_{S}$ and $\mathbf{D}_{T}$ imply that the distributions $\mathrm{Del}_{\delta}(\mathbf{X})$ and $\mathrm{Del}_{\delta}(\mathbf{Y})$ are very close, where $\mathbf{X}$ corresponds to the mixture of Hamming-weight-one strings in $\{0,1\}^{n}$ corresponding to $\mathbf{D}_{S}$ and $\mathbf{Y}$ likewise corresponds to the mixture of Hamming-weight-one strings corresponding to $\mathbf{D}_{T}.$ (In fact, in our setting having $\ell$ matching moments leads to $n^{-\Omega(\ell)}$-closeness in total variation distance, whereas in [DP15] the resulting closeness from $\ell$ matching moments was $2^{-\Omega(\ell)}.$)

We close this subsection by observing that while the results of [Roo00, DP15] were used in a crucial way in subsequent work of Daskalakis et al. [DDS15] to obtain a sample complexity upper bound on learning Poisson binomial distributions, in our context we use these results to obtain a sample complexity lower bound for population recovery. Intuitively, the difference is that in the [DDS15] scenario of learning an unknown Poisson binomial distribution, there is no noise process affecting the samples: the learning algorithm is assumed to directly receive draws from the underlying Poisson binomial distribution being learned. In such a noise-free setting, the existence of a small $\varepsilon$-cover for the space of all Poisson binomial distributions (which is established in [DP15] as a consequence of their moment-matching result) means, at least on a conceptual level, that a learning algorithm “need only search a small space of candidates” to find a high-accuracy hypothesis. In contrast, in our context of deletion-channel noise, our arguments show that it is possible for two underlying true distributions $\mathbf{X},\mathbf{Y}$ over $\{0,1\}^{n}$ to be very different (indeed, to have disjoint supports) but to be such that their deletion-noise-corrupted versions have low-order moments which match each other exactly. In this scenario, the [Roo00, DP15] results can be used to show that the variation distance between the two distributions of noisy samples received by the learner is very small, and this gives a sample complexity lower bound for distinguishing $\mathbf{X}$ and $\mathbf{Y}$ on the basis of such noisy samples.

3 Preliminaries

Notation. Given a nonnegative integer $n$, we write $[n]$ to denote $\{1,\ldots,n\}$. Given integers $a\leq b$ we write $[a:b]$ to denote $\{a,\ldots,b\}$. It will be convenient for us to index a binary string $x\in\{0,1\}^{n}$ using $[0:n-1]$ as $x=(x_{0},\dots,x_{n-1})$. Given a vector $v=(v_{1},\ldots,v_{d})\in\mathbbm{R}^{d}$, we write $\|v\|_{\infty}$ to denote $\max_{i\in[d]}|v_{i}|.$ Given a function $\Delta:A\rightarrow\mathbbm{R}$ over a finite domain $A$, we write $\|\Delta\|_{\infty}=\max_{a\in A}|\Delta(a)|$. Given a polynomial $p$ (which may be univariate or multivariate), we write $\|p\|_{1}$ to denote the sum of magnitudes of $p$’s coefficients. All logarithms and exponents are binary (base 2) unless otherwise specified.

Distributions. We use bold font letters to denote probability distributions and random variables, which should be clear from the context. We write “$\bm{x}\sim\mathbf{X}$” to indicate that random variable $\bm{x}$ is distributed according to distribution $\mathbf{X}$. The total variation distance between two distributions $\mathbf{X}$ and $\tilde{\mathbf{X}}$ over a finite set $\mathcal{X}$ is defined as

[TABLE]

where $\mathbf{X}(x)$ denotes the amount of probability mass that the distribution $\mathbf{X}$ puts on outcome $x$.

Population recovery from the deletion channel. Throughout this paper the parameter $0<$ $\delta<1$ denotes the deletion probability. Given a string $x\in\{0,1\}^{n}$, we write $\mathrm{Del}_{\delta}(x)$ to denote the distribution of a random trace of $x$ after it has been passed through the $\delta$-deletion channel (so the distribution $\mathrm{Del}_{\delta}(x)$ is supported on $\{0,1\}^{\leq n}$). Recall that a random trace $\bm{y}\sim\mathrm{Del}_{\delta}(x)$ is obtained by independently deleting each bit of $x$ with probability $\delta$ and concatenating the surviving bits. 888For simplicity in this work we assume that the deletion probability $\delta$ is known to the learning algorithm. We note that it is possible to obtain a high-accuracy estimate of $\delta$ simply by measuring the average length of traces received from the deletion channel.

We now define the problem of population recovery from the deletion channel that we will study in this paper. In this problem the goal is to learn an unknown target distribution $\mathbf{X}$ supported on at most $\ell$ strings from $\{0,1\}^{n}$. The learning algorithm has access to independent samples, each of which is generated independently by first drawing a string $\bm{x}\sim\mathbf{X}$ and then outputting a trace from $\mathrm{Del}_{\delta}(\bm{x})$. For conciseness we write $\mathrm{Del}_{\delta}(\mathbf{X})$ to denote this distribution. The goal for the learning algorithm is to output with high probability (say at least $0.99$) a hypothesis distribution $\tilde{\mathbf{X}}$ for $\mathbf{X}$ which is $\varepsilon$-accurate in total variation distance: $d_{\mathrm{TV}}(\mathbf{X},\tilde{\mathbf{X}})\leq\varepsilon$. We are interested in the number of samples needed for this learning task in terms of $n$, $\ell$, $\varepsilon$ and $\delta$.

Decks. Given a subset $T=\{t_{1},\ldots,t_{k}\}\subseteq[0:n-1]$ of size $k$ with $t_{1}<\cdots<t_{k}$, and two strings $v\in\{0,1\}^{k}$, $x\in\{0,1\}^{n}$, we say that $v$ matches $x$ at $T$ if $x_{T}=v$, where $x_{T}=(x_{t_{1}},\ldots,x_{t_{k}})\in\{0,1\}^{k}$ denotes the string $x$ restricted to positions in $T$. We say that the number of occurrences of $v$ in $x$ is the number of size-$k$ subsets $T\subseteq[0:n-1]$ such that $v$ matches $x$ at $T$, and we write $\#(v,x)$ to denote this quantity. Given a distribution $\mathbf{X}$ over $\{0,1\}^{n}$, we write $\#(v,\mathbf{X})$ to denote the expected number of occurrences of $v$ in $\bm{x}\sim\mathbf{X}$, i.e.

[TABLE]

Given a string $x\in\{0,1\}^{n}$, we write $\mathsf{D}_{k}(x)$ to denote the (normalized999It will be more convenient for us to use the notion of (normalized) $k$-decks defined here; note that we can recover from it the multi-set of all subsequences of $x$ with length $k$, and vice versa.) $k$-deck of $x$. This is a $2^{k}$-dimensional vector indexed by strings $v\in\{0,1\}^{k}$ such that

[TABLE]

So $\mathsf{D}_{k}(x)$ is a nonnegative vector that sums to $1$. Similarly, for a distribution $\mathbf{X}$ over strings from $\{0,1\}^{n}$, we write $\mathsf{D}_{k}(\mathbf{X})$ to denote the (normalized101010Similarly, the (normalized) $k$-deck here is equivalent to the weighted multi-set version used in the introduction up to a simple rescaling.) $k$-deck of $\mathbf{X}$, given by

[TABLE]

for each $v\in\{0,1\}^{k}$. So $\mathsf{D}_{k}(\mathbf{X})$ is also a $2^{k}$-dimensional nonnegative vector that sums to $1$.

4 Upper bounds for distributions supported on at most $\ell$ strings

Our goal is to prove Theorem 3, which is restated below:

Theorem 3**.**

There is an algorithm $A$ which has the following performance guarantee: For any distribution $\mathbf{X}$ supported over at most $\ell$ strings in $\{0,1\}^{n}$, if $A$ is given

[TABLE]

many samples from $\mathrm{Del}_{\delta}(\mathbf{X})$, then with probability at least 0.99 the algorithm outputs a probability distribution $\tilde{\mathbf{X}}$ supported over at most $\ell$ strings such that $d_{\mathrm{TV}}(\mathbf{X},\tilde{\mathbf{X}})\leq\varepsilon$.

In Section 4.1 we introduce the notion of a restriction, which is a “local view” of a distribution $\mathbf{X}$ confined to a specific subset of coordinates and a specific outcome for those coordinates. We then provide some terminology and prove three useful lemmas about restrictions in Section 4.1. Next in Section 4.2 we describe the algorithm $A$, state our main technical lemma, Lemma 4.6, and use it to prove the correctness of algorithm $A$. We prove Lemma 4.6 in Sections 4.3 and 4.4.

Notational convention. Our argument below involves many integer-valued index variables which take values in a range of different intervals. To help the reader keep track, we will use the following convention (the values $L$ and $m$ will be defined later):

• •

$s,t,s_{1},t_{1},\dots$ will denote an index ranging over $[0:n-1]$;

• •

$j,j_{1},\dots$ will denote an index ranging over $[0:k-1]$;

• •

$a,a^{\prime},a_{1},\dots$ will denote an index ranging over $[L]$;

• •

$b,b^{\prime},b_{1},\dots$ will denote an index ranging over $[0:m]$;

• •

$i,i_{1},\dots$, $\alpha,\alpha_{1},\ldots$ and $\beta,\beta_{1},\ldots$ will denote an index in all other places.

4.1 Restrictions

Let $\mathbf{X}$ be a distribution over strings from $\{0,1\}^{n}$ and let $d\in[n]$ be a parameter (which should be thought of as quite small; we will set $d=O(\log\ell)$ below). Given a size-$d$ subset $T=\{t_{1},\dots,t_{d}\}$ of $[0:n-1]$ with $0\leq t_{1}<\cdots<t_{d}<n$ and a string $c\in\{0,1\}^{d}$, we define

[TABLE]

the probability that a draw of $\bm{x}\sim\mathbf{X}$ matches $c$ in the coordinates of $T$.

Let $\mathbf{X}$ and $\mathbf{Y}$ be two distributions, each supported over at most $\ell$ strings from $\{0,1\}^{n}$. Our first lemma shows that if $d_{\mathrm{TV}}(\mathbf{X},\mathbf{Y})$ is large, then there are a size-$d$ subset $T$ and a string $c\in\{0,1\}^{d}$ with $d=\lfloor\log(2\ell)\rfloor$ such that there is a reasonably big gap between $\mathrm{restrict}(\mathbf{X},T,c)$ and $\mathrm{restrict}(\mathbf{Y},T,c)$.

Lemma 4.1**.**

Let $\mathbf{X}$ and $\mathbf{Y}$ be two distributions, each supported over at most $\ell$ strings from $\{0,1\}^{n}$. Then there exist a size-$d$ subset $T$ of $[0:n-1]$ and a string $c\in\{0,1\}^{d}$ with $d=\lfloor\log(2\ell)\rfloor$ such that

[TABLE]

Proof.

Let $\mathrm{supp}(\mathbf{X})\cup\mathrm{supp}(\mathbf{Y})=\{z^{1},\ldots,z^{\ell^{\prime}}\}$ for some $\ell^{\prime}\leq 2\ell$. For each $i\in[\ell^{\prime}]$, let $p_{i}\geq 0$ be the magnitude of the difference between the probabilities of $z^{i}$ in $\mathbf{X}$ and in $\mathbf{Y}$. Let $\varepsilon=d_{\mathrm{TV}}(\mathbf{X},\mathbf{Y})$. Then by definition we have $\sum_{i}p_{i}=2\varepsilon$. Without loss of generality we assume that $p_{1}\geq\cdots\geq p_{\ell^{\prime}}\geq 0$ and prove the following claim (where we set $p_{\ell^{\prime}+1}=0$ by default for convenience):

Claim 4.2**.**

There exists an $i^{*}\in[\ell^{\prime}]$ such that $p_{i^{*}}\geq\varepsilon/(4\ell)^{\ell^{\prime}}$ and $p_{i^{*}+1}\leq p_{i^{*}}/(4\ell)$.

Proof.

First we notice that $p_{1}\geq\varepsilon/\ell$ given that $\sum_{i}p_{i}=2\varepsilon$ and $\ell^{\prime}\leq 2\ell$. Now given that the $p_{i}$’s are nonnegative, there exists an $i\in[\ell^{\prime}]$ (e.g., by taking $i=\ell^{\prime}$) such that $p_{i+1}\leq p_{i}/(4\ell)$. Take $i^{*}$ to be the smallest such index $i$. Then we have

[TABLE]

by the choice of $i^{*}$ as the smallest such index. As a result, we have

[TABLE]

This finishes the proof of the claim. ∎

Let $i^{*}\in[\ell^{\prime}]$ be the integer given by the claim above, and we consider the first $i^{*}$ strings $z^{1},\ldots,$ $z^{i^{*}}$. Given that $i^{*}\leq\ell^{\prime}\leq 2\ell$, there exist a $d$-subset $T$ of $[0:n-1]$ with $d=\lfloor\log(2\ell)\rfloor$, a string $c\in\{0,1\}^{d}$ and an $i^{\prime}\leq i^{*}$ such that the restriction of $z^{i^{\prime}}$ matches $c$ but the restriction of $z^{i}$ does not match $c$ for any other $i\leq i^{*}$. (This can be achieved by repeatedly selecting a coordinate that splits the remaining strings into two nonempty subsets and setting $c$ to reduce the size by at least half each time.) Using properties of $i^{*}$ given in the claim above, we have

[TABLE]

This finishes the proof of the lemma. ∎

Given two size-$d$ subsets $S=\{s_{1},\ldots,s_{d}\}$ and $T=\{t_{1},\ldots,t_{d}\}$ of $[0:n-1]$ with $s_{1}<\cdots<s_{d}$ and $t_{1}<\cdots<t_{d}$, we say that $S$ is dominated by $T$ if $s_{i}\leq t_{i}$ for every $i\in[d]$. Let $\smash{\Delta:{[0:n-1]\choose d}\rightarrow\mathbbm{R}}$ be a function over size-$d$ subsets of $[0:n-1]$. We use $\mathrm{supp}(\Delta)$ to denote the set of subsets $T$ with $\Delta(T)\neq 0$. We need the following definitions of a cover and a group cover of such a function $\Delta$.

Definition 1** (Covers and group covers).**

We say that a function $\Delta:{[0:n-1]\choose d}\rightarrow\mathbbm{R}$ has an $L$-cover $\{(T_{a},\mathcal{S}_{a}):a\in[L]\}$ for some $L\geq 0$ if

$\mathcal{S}_{1},\ldots,\mathcal{S}_{L}$* form an $L$-way partition of $\mathrm{supp}(\Delta)$; * 2. 2.

$T_{a}\in\mathcal{S}_{a}$* for each $a\in[L]$;* 3. 3.

$\Delta(T)=\Delta(T_{a})$* for every $T\in\mathcal{S}_{a}$; and* 4. 4.

$T_{a}$* is dominated by every $T\in\mathcal{S}_{a}$.*

We refer to the set $T_{a}$ as the anchor set of the collection $\mathcal{S}_{a}$.

Furthermore we say that $\Delta$ has an $(L,q,\lambda)$-group cover if $\Delta$ has an $L$-cover $\{(T_{a},\mathcal{S}_{a}):a\in[L]\}$ and a $q$-way partition of $[L]$ into $A_{1},\ldots,A_{q}$ such that for each $i\in[q]$, for all $a,a^{\prime}\in A_{i}$ we have

[TABLE]

Given distributions $\mathbf{X}$ and $\mathbf{Y}$ over strings from $\{0,1\}^{n}$ and a string $c\in\{0,1\}^{d}$, we write $\Delta_{\mathbf{X},\mathbf{Y},c}$ to denote the function over size-$d$ subsets of $[0:n-1]$ that maps a size-$d$ subset $T$ to

[TABLE]

The second lemma shows that when $d$ and the supports of $\mathbf{X},\mathbf{Y}$ are small, the function $\Delta_{\mathbf{X},\mathbf{Y},c}$ has a small cover for any string $c\in\{0,1\}^{d}$. Taking as an example when $\ell=d=2$ and $\mathrm{supp}(\mathbf{X})=\{x^{1},x^{2}\}$, we have that $\mathrm{restrict}(\mathbf{X},S,c)=\mathrm{restrict}(\mathbf{X},T,c)$ if $x^{1}_{S}=x^{1}_{T}$ and $x^{2}_{S}=x^{2}_{T}$ (note that this is a sufficient but not necessary condition in general). Letting $S=\{s_{1},s_{2}\}$ for some $s_{1}<s_{2}$ and $T=\{t_{1},t_{2}\}$ for some $t_{1}<t_{2}$, this condition can be written equivalently as

[TABLE]

This implies that $\mathrm{restrict}(\mathbf{X},\cdot,c)$, as a function over size-$2$ subsets, has the following combinatorial “rectangular” structure: one can partition indices $t\in[0:n-1]$ into four types 00,01,10,11 according to values of $x^{1}_{t}$ and $x^{2}_{t}$; this induces a partition of all size-${2}$ subsets into $16$ “rectangles,” 111111Strictly speaking, these are not rectangles since we always need to order indices of a subset in ascending order. where $S=\{s_{1}<s_{2}\}$ and $T=\{t_{1}<t_{2}\}$ belong to the same “rectangle” iff the type of $s_{1}$ is the same as that of $t_{1}$ and the type of $s_{2}$ is the same as that of $t_{2}$. It follows that all $T$ in the same “rectangle” share the same value $\mathrm{restrict}(\mathbf{X},T,c)$. We use this observation to obtain a small cover for $\Delta_{\mathbf{X},\mathbf{Y},c}$.

Lemma 4.3**.**

Let $\mathbf{X}$ and $\mathbf{Y}$ be two distributions, each supported over at most $\ell$ strings from $\{0,1\}^{n}$. For any $d\in[n]$ and any string $c\in\{0,1\}^{d}$, $\Delta_{\mathbf{X},\mathbf{Y},c}$ has an $L$-cover for some $L\leq 2^{2d\ell}$.

Proof.

Suppose that $\mathbf{X}$ is supported on $x^{1},\ldots,x^{\ell^{\prime}}$ and $\mathbf{Y}$ is supported on $y^{1},\ldots,y^{\ell^{\prime\prime}}$ with $\ell^{\prime},\ell^{\prime\prime}\leq\ell$. We say an index $t\in[0:n-1]$ is of type-$(u,v)$, where $u\in\{0,1\}^{\ell^{\prime}}$ and $v\in\{0,1\}^{\ell^{\prime\prime}}$, if

[TABLE]

This allows us to classify size-$d$ subsets of $[0:n-1]$ into at most $\smash{(2^{\ell^{\prime}+\ell^{\prime\prime}})^{d}\leq 2^{2d\ell}}$ many equivalence classes: $S\sim T$ if $S=\{s_{1},\ldots,s_{d}\}$ with $s_{1}<\cdots<s_{d}$ and $T=\{t_{1},\ldots,t_{d}\}$ with $t_{1}<\cdots<t_{d}$ are such that $s_{i}$ and $t_{i}$ are of the same type for all $i\in[d]$.

Let $\mathcal{S}_{a}$ be a nonempty equivalence class of $\sim$ such that $S=\{s_{1},\ldots,s_{d}\}\in\mathcal{S}_{a}$ if $s_{1}<\cdots<s_{d}$ and $s_{i}$ has type-$(u^{(i)},v^{(i)})$ for each $i\in[d]$. It follows from the definition of $\sim$ that all $S\in\mathcal{S}_{a}$ have the same $\mathrm{restrict}(\mathbf{X},S,c)$ and $\mathrm{restrict}(\mathbf{Y},S,c)$, and hence the same value of $\Delta_{\mathbf{X},\mathbf{Y},c}(S)$. Moreover, we let $T_{a}=\{t_{1},\ldots,t_{d}\}$ be the following set: $t_{1}$ is the smallest index of type-$(u^{(1)},v^{(1)})$ and for each $i$ from $2$ to $d$, $t_{i}$ is the smallest index that is larger than $t_{i-1}$ and has type-$(u^{(i)},v^{(i)})$. Because $\mathcal{S}_{a}$ is nonempty, $T_{a}$ is well defined and it is easy to verify that $T_{a}$ is dominated by every $S\in\mathcal{S}_{a}$. As a result, $\Delta_{\mathbf{X},\mathbf{Y},c}$ has the following $L$-cover:

[TABLE]

for some $L\leq 2^{2d\ell}$. This finishes the proof of the lemma. ∎

The last lemma shows that the function $\Delta_{\mathbf{X},\mathbf{Y},c}$ actually has an $(L,q,\lambda)$-group cover, for some parameters $L\leq 2^{2d\ell}$, $q\leq\ell$ and $\lambda\leq\ell^{O(\ell)}$.

Lemma 4.4**.**

Let $\mathbf{X}$ and $\mathbf{Y}$ be two distributions, each supported over at most $\ell$ strings from $\{0,1\}^{n}$. For any $d\in[n]$ and $c\in\{0,1\}^{d}$, $\Delta_{\mathbf{X},\mathbf{Y},c}$ has an $(L,q,\ell^{O(\ell)})$-group cover for some $L\leq 2^{2d\ell}$ and $q\leq\ell$.

Proof.

First we apply Lemma 4.3 to obtain an $L$-cover $\{(T_{a},\mathcal{S}_{a}):a\in[L]\}$ of $\Delta:=\Delta_{\mathbf{X},\mathbf{Y},c}$ for some $L\leq 2^{2d\ell}$. It suffices to show that the $L$ positive numbers $|\Delta(T_{a})|$, $a\in[L]$, can be divided into at most $\ell$ groups such that any two in the same group have the ratio bounded from above by $\ell^{O(\ell)}$.

Let $p_{1},\ldots,p_{\ell^{\prime}}>0$ be probabilities of strings in $\mathbf{X}$ for some $\ell^{\prime}\leq\ell$ and $q_{1},\ldots,q_{\ell^{\prime\prime}}>0$ be probabilities of strings in $\mathbf{Y}$ for some $\ell^{\prime\prime}\leq\ell$. The observation is that every number $|\Delta(T_{a})|$ is a linear form over the $p_{i}$’s and $q_{i}$’s with coefficients $-1,0$ or $1$. This motivates the following claim:

Claim 4.5**.**

Let $u_{1},\ldots,u_{g}>0$ be $g$ (not necessarily distinct) positive numbers. Let

[TABLE]

Then there cannot exist $g+1$ numbers $v_{1},\ldots,v_{g+1}$ in $V$ satisfying $v_{g+1}>\cdots>v_{1}$ and

[TABLE]

Proof.

Assume for a contradiction that such $g+1$ numbers $v_{1},\ldots,v_{g+1}$ exist in $V$ and let

[TABLE]

where $c_{i,j}\in\{-1,0,1\}$ for each $i\in[g+1]$. Given that these are $g+1$ many $g$-dimensional vectors $c_{i}$ $=(c_{i,1},\ldots,c_{i,g})$, let $i^{*}\leq g+1$ be the smallest integer such that $c_{i^{*}}$ can be written as a linear combination of $c_{1},\ldots,c_{i^{*}-1}$: $c_{i^{*}}=\alpha_{1}c_{1}+\cdots+\alpha_{i^{*}-1}c_{i^{*}-1}$, which implies that

[TABLE]

We show below that the magnitude of coefficients $\alpha_{1},\ldots,\alpha_{i^{*}-1}$ is relatively small, which leads to a contradiction because we assumed that $v_{i^{*}}$ is much bigger than $v_{i^{*}-1},\ldots,v_{1}$.

To see this, note that $(\alpha_{1},\ldots,\alpha_{i^{*}-1})$ is the solution to a $(i^{*}-1)\times(i^{*}-1)$ linear system $Ax=b$ where $A$ is a $\{-1,0,1\}$-valued $(i^{*}-1)\times(i^{*}-1)$ full-rank matrix and $b$ is a $\{-1,0,1\}$-valued vector. (In more detail, one can take $A$ to be a full-rank $(i^{*}-1)\times(i^{*}-1)$ submatrix of the matrix that consists of $c_{1},\ldots,c_{i^{*}-1}$ as columns and take the vector $b$ to be the corresponding entries of $c_{i^{*}}$.) It follows from Cramer’s rule that each entry of $A^{-1}$ has magnitude at most $(i^{*}-1)!$ and thus, each entry of $A^{-1}b$ has absolute value at most $(i^{*}-1)\cdot(i^{*}-1)!<i^{*}!\leq(g+1)!$ This contradicts with (6) and the assumption that $v_{1}<\ldots<v_{i^{*}-1}\leq v_{i^{*}}/(g+2)!$. ∎

Claim 4.5 gives us the following procedure to partition $[L]$ into $A_{1},\ldots,A_{q}$ for some $q\leq\ell$:

Set $i=1$ and $\mathcal{L}=[L]$. 2. 2.

While $\mathcal{L}$ is nonempty do 3. 3.

Let $v$ be the smallest $|\Delta(T_{a})|$, $a\in\mathcal{L}$. 4. 4.

Remove from ${\cal L}$ and add to $A_{i}$ every $a\in\mathcal{L}$ with $|\Delta(T_{a})|\leq(2\ell+2)!\cdot v$, and increment $i$.

It follows from Claim 4.5 that when $\mathcal{L}$ becomes empty at the end, the number of $A_{i}$’s we created can be no more than $\ell$. Furthermore, every $a$ and $a^{\prime}$ that belong to the same $A_{i}$ have the ratio of $|\Delta(T_{a})|$ and $|\Delta(T_{a^{\prime}})|$ bounded by $(2\ell+2)!=\ell^{O(\ell)}.$ This finishes the proof of the lemma. ∎

4.2 Main Algorithm

We start with an algorithm, based on dynamic programming, for estimating the $k$-deck of a distribution $\mathbf{X}$ over $\{0,1\}^{n}$.

Theorem 4**.**

Let $k\in[n]$. There is an algorithm with the following performance guarantee: for any distribution $\mathbf{X}$ over strings in $\{0,1\}^{n}$, if the algorithm is given

[TABLE]

many samples from $\mathrm{Del}_{\delta}(\mathbf{X})$ then with probability at least $0.99$ the algorithm outputs a nonnegative $2^{k}$-dimensional vector $Q$ with $\|Q-\mathsf{D}_{k}(\mathbf{X})\|_{\infty}\leq\xi.$ Its running time is $2^{k}M\cdot\mathrm{poly}(n)$.

Proof.

Let $x^{1},\ldots,x^{p}$ be the support of $\mathbf{X}$. Then for each string $v\in\{0,1\}^{k}$, we have

[TABLE]

The first equation is because for a given size-$k$ subset $S\subseteq[0:n-1]$ of indices at which $v$ matches $x^{i}$, all of the positions in $S$ “survive” into a string $\bm{z}\sim\mathrm{Del}_{\delta}(x^{i})$ with probability exactly $(1-\delta)^{k}.$

As a result, it suffices to estimate $\mathop{{\bf E}\/}[\#(v,\bm{z})]$ to additive accuracy $\pm\xi(1-\delta)^{k}{n\choose k}$ for every string $v\in\{0,1\}^{k}$. For any fixed string $v\in\{0,1\}^{k}$, by a standard Chernoff bound, using

[TABLE]

samples the empirical estimate of $\operatorname{{\bf E}}[\#(v,\bm{z})]$ will have the desired additive $\xi(1-\delta)^{k}{n\choose k}$ accuracy except with failure probability $0.01/2^{k}$. The success probability of 0.99 follows from union bound.

The running time of the algorithm uses the following simple observation: given $z\in\{0,1\}^{n^{\prime}}$ and $v\in\{0,1\}^{k}$, there is a $\mathrm{poly}(n^{\prime},k)$-time procedure that computes $\#(v,z).$ The procedure works by straightforward dynamic programming: For each $j\in[0:k-1]$ and $i\in[0:n^{\prime}-1]$, the algorithm maintains a count of the number $\#(v_{0}\ldots v_{j},z_{0}\ldots z_{i})$. This then implies that the running time of the overall algorithm is $M\cdot 2^{k}\cdot\mathrm{poly}(n)$. This finishes the proof of the lemma. ∎

We prove the following main technical lemma in Sections 4.3 and 4.4. Intuitively, this lemma says that if the total variation distance between $\mathbf{X}$ and $\mathbf{Y}$ is not too small, then for a suitable (not too large) value of $k^{\ast},$ the distance between the $k^{\ast}$-decks of $\mathbf{X}$ and $\mathbf{Y}$ also cannot be too small.

Lemma 4.6**.**

Let $\ell$ be a positive integer with $\ell\leq\log n$. Let $\mathbf{X}$ and $\mathbf{Y}$ be two distributions, each supported over at most $\ell$ strings from $\{0,1\}^{n}$. Then there is a positive integer

[TABLE]

such that

[TABLE]

We now present our algorithm $A$ and use Lemma 4.6 to prove Theorem 3:

Proof of Theorem 3.

The bound (5) we aim for holds trivially when $\ell\geq\log n$. To see this, we first notice that when $\ell\geq\log n$, the sample complexity bound (5) we aim for is at least

[TABLE]

With $(1/(1-\delta))^{n}$ samples from $\mathrm{Del}_{\delta}(\mathbf{X})$, we expect to see a full string of length $n$ where no bits are deleted and we know that such a string is drawn directly from $\mathbf{X}$. This means that, with (8) many samples, we receive $\text{poly}(\ell)/\varepsilon^{2}$ draws from $\mathbf{X}$ with high probability. When the latter happens, the empirical estimation $\tilde{\mathbf{X}}$ of $\mathbf{X}$ satisfies $d_{\mathrm{TV}}(\mathbf{X},\tilde{\mathbf{X}})\leq\varepsilon$ with high probability. This allows us to focus on the case when $\ell\leq\log n$ in the rest of the proof (so Lemma 4.6 applies).

Let $\varepsilon$ be the total variation distance we aim for in Theorem 3. Let $k^{*}$ be the parameter in (7). Let $\xi$ be a parameter to be specified later. By Theorem 4, the algorithm $A$ can first use

[TABLE]

samples to obtain an estimate $Q$ of $\mathsf{D}_{k^{*}}(\mathbf{X})$ such that

[TABLE]

and it succeeds in obtaining such an estimate with probability at least $0.99$.

With $Q$ in hand the algorithm $A$ computes $\|Q-\mathsf{D}_{k^{*}}(\mathbf{Y})\|_{\infty}$ for every distribution $\mathbf{Y}$ supported on at most $\ell$ strings such that the probability of each string in $\mathbf{Y}$ is an integer multiple of $\xi/\ell$. Finally the algorithm outputs the distribution $\mathbf{X}^{*}$ that minimizes the distance (breaking ties arbitrarily).

We show that when $Q$ satisfies (10), $\mathbf{X}^{*}$ must be close to $\mathbf{X}$. We start with a simple observation that one can round $\mathbf{X}$ to get a distribution $\mathbf{X}^{\prime}$ in which the probability of each string is an integer multiple of $\xi/\ell$ and $d_{\text{TV}}(\mathbf{X},\mathbf{X}^{\prime})\leq\xi$. This can be done by rounding the probability of every string except one to the nearest multiple of $\xi/\ell$ and setting the last probability as required so that the total probability is 1. We have

[TABLE]

By definition of $\mathbf{X}^{*}$ and $\mathbf{X}^{\prime}$, we have $\|Q-\mathsf{D}_{k^{*}}(\mathbf{X}^{*})\|_{\infty}\leq\|Q-\mathsf{D}_{k^{*}}(\mathbf{X}^{\prime})\|_{\infty}\leq 2\xi$. As a result,

[TABLE]

It follows from Lemma 4.6 that

[TABLE]

Finally we choose $\xi$ so that the RHS becomes $\varepsilon$. The number of samples needed in (9) becomes

[TABLE]

This finishes the proof of Theorem 3. ∎

We use the following two lemmas to prove Lemma 4.6. They are proved in Section 4.3 and 4.4.

Lemma 4.7**.**

Let $d,q,L$ and $\lambda$ be positive integers satisfying

[TABLE]

Let $\smash{\Delta:{[0:n-1]\choose d}\rightarrow\mathbbm{R}}$ be a function that is not identically zero and has an $(L,q,\lambda)$-group cover. Let $m=d(n-1)L^{2}$. Then there exists a $d$-variate polynomial $\phi$ with degree at most $O(\sqrt{m}\cdot\log^{{4q+1}}m)$ and $\|\phi\|_{1}=\exp(O(\sqrt{m}\cdot\log^{{4q+3}}m))$ such that

[TABLE]

We note that the following lemma holds for any two distributions $\mathbf{X},\mathbf{Y}$ over $\{0,1\}^{n}$ regardless of their support size.

Lemma 4.8**.**

Let $d,k\in[n]$ with $k\geq d$. Let $\mathbf{X},\mathbf{Y}$ be distributions each supported over strings from $\{0,1\}^{n}$. Then for any string $c\in\{0,1\}^{d}$ and $d$-variate polynomial $\phi$ of degree at most $k-d$,

[TABLE]

Proof of Lemma 4.6.

Let $\mathbf{X}$ and $\mathbf{Y}$ be two distributions each supported over at most $\ell$ strings from $\{0,1\}^{n}$. It then follows from Lemma 4.1 and Lemma 4.4 that there exists a string $c\in\{0,1\}^{d}$ with $d=\lfloor\log(2\ell)\rfloor$ such that $\Delta:=\Delta_{\mathbf{X},\mathbf{Y},c}$ satisfies $\|\Delta\|_{\infty}\geq d_{\mathrm{TV}}(\mathbf{X},\mathbf{Y})/\ell^{O(\ell)}$ and has an $(L,q,\lambda)$-group cover for some $L\leq 2^{2d\ell}$, $q\leq\ell$, and $\lambda=\ell^{O(\ell)}.$ As we assumed that $\ell\leq\log n$, both $d$ and $q$ are at most $\log n$ and $L,\lambda\leq\ell^{O(\ell)}\leq(\log n)^{O(\log n)}$ (so Lemma 4.7 applies).

Let $m=d(n-1)L^{2}$ and $\phi$ be the polynomial given in Lemma 4.7. Let $k^{*}=\deg(\phi)+d$ (we set $k=k^{*}$ in Lemma 4.8; the choice of $k^{*}$ ensures that $\deg(\phi)\leq k^{*}-d$ as required in Lemma 4.8) with

[TABLE]

Combining Lemma 4.7 and Lemma 4.8, we have

[TABLE]

The lemma follows from the fact that $\|\Delta\|_{\infty}\geq d_{\mathrm{TV}}(\mathbf{X},\mathbf{Y})/\ell^{O(\ell)}$.∎

4.3 Proof of Lemma 4.7

Let $\smash{\Delta}$ be a function over $d$-subsets of $[0:n-1]$ that is not identically zero and has an $(L,q,\lambda)$-group cover $\{(T_{a},\mathcal{S}_{a}):a\in[L]\}$ with a $q$-way partition $A_{1},\ldots,A_{q}$ of $[L]$. We start with a high-level description of the $d$-variate polynomial $\phi$.

To evaluate $\phi$ on a tuple $(t_{1},\ldots,t_{d})$, we first project $(t_{1},\ldots,t_{d})$ onto a line along the direction of $(w_{1},\ldots,w_{d})$ for some relatively small positive integers $w_{1},\ldots,w_{d}$ to be specified later, and then apply a univariate polynomial $f(\cdot)$ on the image of the projection. In other words $\phi$ takes the form

[TABLE]

for some positive integers $w_{1},\ldots,w_{d}\in[L^{2}]$. We give details below.

4.3.1 The projection

Let $m=d(n-1)L^{2}$ and let $w$ be the following function from size-$d$ subsets of $[0:n-1]$ to $[0:m]$:

[TABLE]

So $w$ is the projection function that maps a size-$d$ subset $T$ of $[0:n-1]$ (or equivalently, a sorted $d$-tuple of distinct values from $[0:n-1]$) to a location on the real line. Claim 4.9 implies that there exist $w_{1},\ldots,w_{d}\in[L^{2}]$ such that all anchor sets in the $L$-cover are mapped to distinct locations.

Claim 4.9**.**

If $w_{1},\ldots,w_{d}$ are drawn independently and uniformly at random from $[L^{2}]$ then $w(T_{a})$ $\neq w(T_{a^{\prime}})$ for all $a\neq a^{\prime}\in[L]$ with probability at least $1/2$.

Proof.

Let $S=\{s_{1},\ldots,s_{d}\}$ and $T=\{t_{1},\ldots,t_{d}\}$ denote two size-$d$ subsets of $[0:n-1]$ that satisfy $s_{1}<$ $\cdots<s_{d}$, $t_{1}<\cdots<t_{d}$ and $S\neq T$. Then the probability that $w(S)=w(T)$ equals

[TABLE]

As $(s_{1},\dots,s_{d})\neq(t_{1},\dots,t_{d})$, one of the $d$ quantities $s_{i}-t_{i}$ is nonzero; say without loss of generality $s_{1}\neq t_{1}.$ Fixing any outcomes of random draws of $w_{2},\dots,w_{d}$, there is a unique outcome of $w_{1}$ which would result in the equation in (12), and the probability that $w_{1}$ takes this particular outcome is either $1/L^{2}$ or zero (if it is not in $[L^{2}]$). As a result, the probability in (12) is at most $1/L^{2}$, and the claim follows from a union bound over ${L\choose 2}$ events. ∎

We fix such a tuple $w_{1},\ldots,w_{d}\in[L^{2}]$ that satisfies Claim 4.9 for the rest of the proof.

4.3.2 The univariate polynomial

Now we move to the more difficult part of choosing the univariate polynomial $f$ in (11).

A useful tool. A key tool for our construction of $f$ is a univariate polynomial $h$ with several useful properties described below. Figure 1 gives a schematic representation of the key upper bounds on $|h(b)|$ provided by item (2) in Lemma 4.10.

Lemma 4.10**.**

There is a univariate polynomial $h$ with the following properties:

$h$* has degree $O(\sqrt{m}\log m)$.* 2. 2.

$h(0)=1$* and for each $b\in[m]$,*

[TABLE] 3. 3.

$h$* satisfies $\|h\|_{1}\leq\exp(O(\sqrt{m}\log m))$.*

Our construction of the polynomial $h$ is based on the Chebyshev polynomial and builds on an earlier construction due to Borwein et al. [BEK99]. We prove Lemma 4.10 in Appendix A, and we explain the role that $h$ plays in the construction of our desired univariate polynomial $f$ under the heading “The high-level idea” below, after first providing some useful preliminary explanation.

Given that our polynomial $\phi$ takes the form of (11), the crucial quantity whose magnitude we are trying to lower bound, namely

[TABLE]

(recall the LHS of Lemma 4.7), can be written as

[TABLE]

where $\Gamma:[0:m]\rightarrow\mathbbm{R}$ is a function that is defined using $\Delta$ as follows:

[TABLE]

where the sum is over all $d$-subsets $T$ of $[0:n-1]$.

To better understand $\Gamma$, we use the $(L,q,\lambda)$-group cover of $\Delta$ to introduce two new sequences $\tau_{0},\ldots,\tau_{r}$ and $m_{0},\ldots,m_{r}$, for some value $r\in[0:q-1]$ that is defined below. We start with some notation. For each $i\in[q]$, we let $\mathcal{G}_{i}=\cup_{a\in A_{i}}\mathcal{S}_{a}$ and refer to $\mathcal{G}_{i}$ as group $i$. We refer to the $T_{a}$ with the smallest $w(T_{a})$ among all $a\in A_{i}$ as the anchor of group $i$ and denote it by $V_{i}$. (By Claim 4.9, each group has a unique anchor and we have $w(T)>w(V_{i})$ for all $T\in\mathcal{G}_{i}$ other than $V_{i}$.) We let $v_{i}=|\Delta(V_{i})|$ and $\kappa_{i}=w(V_{i})$, so $\kappa_{i}$ is the location that the anchor $V_{i}$ of $\mathcal{G}_{i}$ is projected to. By the definition of an $(L,q,\lambda)$-group cover and Claim 4.9, we have that each $v_{i}>0$, the $\kappa_{i}$’s are distinct,

[TABLE]

Now we are ready to define $r$ and the two sequences. See Figure 2 for an illustration of these sequences. First we let $\tau_{0}=\max_{i\in[q]}v_{i}$ and also let $m_{0}\in[0:m]$ denote the smallest $\kappa_{i}$ (among all groups $i\in[q]$) with $v_{i}=\tau_{0}$. We are done and the value of $r$ is 0 if no $\kappa_{i}$ is smaller than $m_{0}$ (i.e. the anchor of every other group is projected to a larger location value than $m_{0}$); otherwise, we let $\tau_{1}<\tau_{0}$ be the largest value of $v_{i}$ over those $i\in[q]$ that have $\kappa_{i}<m_{0}$ and also let $m_{1}<m_{0}$ be the smallest $\kappa_{i}$ such that $v_{i}=\tau_{1}$. We are done and the value of $r$ is 1 if no $\kappa_{i}$ is smaller than $m_{1}$ (i.e. every other group anchor is projected to a larger location value than $m_{1}$); otherwise we repeat the process. Continuing in this way, at the end we obtain two sequences:

[TABLE]

for some value $r\in[0:q-1]$. We say that $(\tau_{0},\dots,\tau_{r})$ is the $\tau$-step-sequence and that $(m_{0},\dots,m_{r})$ is the $m$-step-sequence for $\Gamma$.

The high-level idea. Before entering into further details we give intuition for the polynomial $f$. Looking ahead to (23), the polynomial $f$ is essentially a translation of the polynomial $h$ depicted in Figure 1, i.e. $f(x)$ is essentially $h(x-m_{\alpha})$ for some $\alpha\in[0:r]$.121212The exponent of $h$ in the exact definition of our $f$ given in Equation (23) is needed for technical reasons that are not important for this intuitive explanation. Recalling the key properties of $h$, we see that

• •

$f(m_{\alpha})=1.$

• •

$|f(b)|$ is “very small” for $b>m_{\alpha}$; and

• •

$|f(b)|$ is “not too large” for $b<m_{\alpha}$.

The crux of our analysis below is to establish that there is a suitable value $m_{\alpha}$ in the $m$-step-sequence which is such that the magnitude of the single summand $f(m_{\alpha})\cdot\Gamma(m_{\alpha})$ in (13) is greater than the contribution of all other summands in (13).

To gain some intuition for why this is the case, let us pretend that instead of the $\Gamma$ being defined as in (13), the definition of $\Gamma$ instead only took a sum over the $q$ anchors $V_{1},\dots,V_{q}$ of the $q$ groups $\mathcal{G}_{1},\ldots,\mathcal{G}_{q}$ (i.e. $\Gamma$ is supported on $\kappa_{i}$, $i\in[q]$, with $|\Gamma(\kappa_{i})|=v_{i}$). Of course this is not actually the case since each group $\mathcal{G}_{i}$ in general contains many more sets than just its anchor $T_{i}$, but it turns out that the effect of other sets in $\mathrm{supp}(\Delta)$ will only cost us some extra $n^{d}\lambda$ factors in the analysis (corresponding to the $n^{d}\lambda$ factors in properties (ii) and (iii) of $\Gamma_{0},\dots,\Gamma_{r}$ as described below, where $\smash{n^{d}\geq{n\choose d}}$ just serves as a bound for the number of size-$d$ subsets) which turn out to be manageable.

In this hypothetical scenario the only nonzero values of $\Gamma(b)$ that would enter the picture would be the $v_{i}$ values at locations $\kappa_{i}$, $i\in[q]$, which are the heights of the bars in Figure 2. The desired $m_{\alpha}$ could then be identified as follows:

• •

We proceed in an inductive fashion. For each $p\in[0:r]$, we show that there is a choice of $\alpha\in[0:p]$ such that, by setting $f(x)=h(x-m_{\alpha})$, the value of $|f(m_{\alpha})\cdot\Gamma(m_{\alpha})|$ outweighs $|f(b)\cdot\Gamma(b)|$ for every other $b\in[m_{p}:m]$. The choice of $\alpha$ at the end of the induction when $p$ reaches $r$ gives us the desired location $m_{\alpha}$ for the translation of $h$ to define $f$.

• •

The base case when $p=0$ is trivial by setting $\alpha=0$ and $f(x)=h(x-m_{0})$. Here we have that $|f(m_{0})\cdot\Gamma(m_{0})|$ outweighs $|f(b)\cdot\Gamma(b)|$ for all $b>m_{0}$ because $|\Gamma(m_{0})|=\tau_{0}\geq|\Gamma(b)|$ by the definition of our step-sequences and the fact that $f(m_{0})=1$ is “much larger” than $|f(b)|$ for $b>m_{0}$.

• •

Next we move to $p=1$, and now we need to take $\Gamma(b)$, $b\in[m_{1}:m_{0}-1]$, into consideration. To this end we compare $\tau_{0}/\tau_{1}$ with $\exp(\sqrt{m_{0}-m_{1}})$ and consider the following two cases.

• –

If $\tau_{0}/\tau_{1}$ is larger then we can keep $\alpha=0$ and $f(x)=h(x-m_{0})$ because $|f(m_{0})\cdot\Gamma(m_{0})|$ outweighs $|f(m_{1})\cdot\Gamma(m_{1})|$ (since $\Gamma(m_{1})=\tau_{1}$ and $f(m_{1})$ is roughly131313This is not entirely precise because in (2) of Lemma 4.10 there is indeed an extra factor of $\log m$ in the exponent on the left side of [math]; overcoming this factor of $\log m$ is the reason why we end up with the exponent as in (23). $\exp(\sqrt{m_{0}-m_{1}})$) as well as $|f(b)\cdot\Gamma(b)|$ for all $b\in[m_{1}+1:m_{0}-1]$ (since $|f(m_{1})|>|f(b)|$ and by the definition of our step-sequences, $|\Gamma(m_{1})|\geq|\Gamma(b)|$). By the inductive hypothesis we also know that $|f(m_{0})\cdot\Gamma(m_{0})|$ outweighs $|f(b)\cdot\Gamma(b)|$ for all $b>m_{0}$.

• –

Otherwise (if $\tau_{1}$ is larger than $\tau_{0}/\exp(\sqrt{m_{0}-m_{1}})$) we show that setting $\alpha=1$ and $f(x)=h(x-m_{1})$ works. On the one hand, $|f(m_{1})\cdot\Gamma(m_{1})|$ outweighs $|f(b)\cdot\Gamma(b)|$ for $b\in[m_{1}+1:m_{0}-1]$ since $|\Gamma(b)|\leq|\Gamma(m_{1})|$ by the definition of our step-sequences and the fact that $f(m_{1})=1$ is “much larger” than $|f(b)|$ (similar to the base case). On the other hand, $|f(m_{1})\cdot\Gamma(m_{1})|=\tau_{1}$ outweighs $|f(m_{0})\cdot\Gamma(m_{0})|=|f(m_{0})|\cdot\tau_{0}$ (since $|f(m_{0})|$ is, roughly speaking, $\exp(-\sqrt{m_{0}-m_{1}})$) as well as $|f(b)\cdot\Gamma(b)|$ for $b>m_{0}$ (since $|f(m_{0})|>|f(b)|$ and $|\Gamma(m_{0})|\geq|\Gamma(b)|$ so the contribution from $b$ is smaller than that from $m_{0}$).

• •

Continuing in this fashion, we show that, if $\alpha$ is the choice for some $p\in[0:r-1]$, then for $p+1$ we can either keep the same choice of $\alpha$ or move $\alpha$ to $p+1$, depending on the result of a similar comparison between $\tau_{\alpha}/\tau_{p+1}$ and $\exp(\sqrt{m_{\alpha}-m_{p+1}})$. This finishes the induction.

The above reasoning is formalized in the statement and proof of the (crucial) Lemma 4.11, which additionally has to deal with the complication that it must address the real scenario rather than the hypothetical simplification considered in the informal description above.

Now we turn to the details. For each $b\in[0:m]$, we let

[TABLE]

For each $p\in[0:r]$ let $\Gamma_{p}$ denote the following function on $[0:m]$:

[TABLE]

In words, the value of $\Gamma_{p}$ evaluated at a location value $b$ is obtained as follows: for each group $\mathcal{G}_{i}$ for which the location $\kappa_{i}$ that the anchor set $V_{i}$ is projected to is at least $m_{p}$, we sum the value of $\Delta(T)$ over all $T\in\mathcal{G}_{i}$ which are mapped by the projection function $w$ to the location $b$.

Example 5**.**

In Figure 2, only $\mathcal{G}_{i}$’s that correspond to shaded diamonds are considered in $\Gamma_{1}$.

We have the following properties from our choices of $\tau_{i}$’s and $m_{i}$’s:

1. (i)

$\Gamma=\Gamma_{r}$. This is because every location $\kappa_{i}$ is at least $m_{r}$. 2. (ii)

$\Gamma_{0}$ is such that $\Gamma_{0}(b)=0$ for all $b<m_{0}$, $|\Gamma_{0}(m_{0})|=\tau_{0}$, and

[TABLE]

for all $b>m_{0}$. (The last bound holds just because there are at most $n^{d}$ many size-$d$ subsets and the maximum value of $|\Delta(T)|$ on any $T$ contributing to the sum $\Gamma_{0}(b)$ is at most $\lambda\tau_{0}$.) 3. (iii)

Generalizing the previous property, for each $p\in[r]$, $\Gamma_{p}(b)=0$ for $b<m_{p}$, $|\Gamma_{p}(m_{p})|=\tau_{p}$ and

[TABLE]

for all $b>m_{p}$ (since the maximum magnitude of $\Delta(T)$ on any subset $T$ contributing to the sum $\Gamma_{p}(b)$ but not to $\Gamma_{p-1}(b)$ is at most $\lambda\tau_{p}$).

We prove the following crucial lemma, Lemma 4.11, concerning $\Gamma_{p}$ by induction on $p$. Intuitively, the lemma states that for each $p$ there is a suitable index $\alpha\leq p$ (so $m_{\alpha}\geq m_{p}$) such that (a) the magnitude of $\Gamma_{p}(m_{\alpha})$ is not too small compared to $\tau_{0}$ (this is given by (15)); (b) for locations $b>m_{\alpha}$ the magnitude of $\Gamma_{p}(b)$ is not too large compared to the magnitude of $\Gamma_{p}(m_{\alpha})$ (this is given by (16)); and (c) for locations $b$ between $m_{p}$ and $m_{\alpha}$ the magnitude of $\Gamma_{p}(b)$ is small compared to the magnitude of $\Gamma_{p}(m_{\alpha})$ (this is given by (17)). In all three places the meaning of “small” or “large” is specified by a second parameter $\beta$ which can grow slowly with $p$.

Lemma 4.11**.**

Assume that $d\leq\log n$ and $\lambda\leq(\log n)^{O(\log n)}$. Then for each $p\in[0:r]$ there are two parameters $\alpha_{p}\in[0:p]$ and $\beta_{p}\in[0:4p+3]$ (letting $\alpha$ denote $\alpha_{p}$ and $\beta$ denote $\beta_{p}$ below for convenience) such that

[TABLE]

and every index $b\in[m_{p}:m]$ satisfies

If $b\geq m_{\alpha}$, then

[TABLE] 2. 2.

If $m_{p}\leq b<m_{\alpha}$, then

[TABLE]

Proof.

The base case when $p=0$ follows from properties of $\Gamma_{0}$ by setting $\alpha_{0}=0$ and $\beta_{0}=3$ since $d\leq\log n$ and $\lambda\leq(\log n)^{O(\log n)}$ imply that $n^{d}\lambda\ll\exp(\log^{3}m)$.

For the induction step, we assume that the claim holds for some $p\in[0:r-1]$ with parameters $\alpha=\alpha_{p}\in[0:p]$ and $\beta=\beta_{p}\in[0:4p+3]$, and now we prove it for $p+1$. We consider two cases:

[TABLE]

For the case of (18), we verify that the claim holds for $p+1$ by setting $\alpha_{p+1}=\alpha$ and $\beta_{p+1}=\beta$. First, by property (iii) and (18) we have

[TABLE]

As a result, (15) holds for $p+1$ since

[TABLE]

by the inductive hypothesis of (15) for $p$. For each $b\geq m_{\alpha}$, compared with the inductive hypothesis (16) for $p$, the LHS of (16) for $p+1$ goes up (additively) by at most $n^{d}\lambda\tau_{p+1}$ (by property (iii)), but it follows from (20) that the RHS of (16) for $p+1$ goes up multiplicatively by a factor of $2(1-o_{n}(1))$. Since by (18) we have $|\Gamma_{p}(m_{\alpha})|\gg n^{d}\lambda\tau_{p+1}$, (16) also holds for $p+1$. Similarly for each $m_{p}\leq b<m_{\alpha}$, compared to the inductive hypothesis (17) for $p$, again by property (iii) the LHS of (17) for $p+1$ goes up (additively) by at most

[TABLE]

while the RHS goes up (additively) by at least

[TABLE]

So (17) still holds for $p+1$. Finally, for each $m_{p+1}\leq b<m_{p}$, we have by property (iii) that

[TABLE]

So (17) also holds for such ${b}$’s. This finishes the induction step assuming (18).

We now consider (19) and show that the claim holds for $p+1$ by setting $\alpha_{p+1}=p+1$ and $\beta_{p+1}$ $=\beta+4\leq 4p+7$. First we prove that (15) holds for $p+1$. Combining (19), the inductive hypothesis (15) for $p$, and that $|\Gamma_{p+1}(m_{p+1})|=\tau_{p+1}$, we have

[TABLE]

where the last inequality used the assumptions on $d$ and $\lambda$ so that $n^{d}\lambda\ll\exp(\log^{3}m)$. Next, we consider each $b\geq m_{p+1}$ and show that (16) holds for $p+1$ (note that the conditions for (17) never occur in this case since we set $\alpha_{p+1}=p+1$ and hence $m_{\alpha_{p+1}}=m_{p+1}$):

[TABLE]

This is trivial for $m_{p+1}\leq b<m_{p}$ as $|\Gamma_{p+1}(b)|\leq n^{d}\lambda\tau_{p+1}$ by property (iii) and $n^{d}\lambda\ll\exp(\log^{3}m)$. For each $b:m_{p}\leq b<m_{\alpha}$, we have by the inductive hypothesis (17) that

[TABLE]

Combining (19) with (21), we have

[TABLE]

Using $\sqrt{m_{\alpha}-m_{p+1}}-\sqrt{m_{\alpha}-b}\leq\sqrt{b-m_{p+1}}$, we have

[TABLE]

using $n^{d}\lambda\ll\exp(\log^{3}m)$. The last case for (16) is that $b\geq m_{\alpha}$. By the inductive hypothesis,

[TABLE]

Combining (19) with (22), we have

[TABLE]

Using $\sqrt{b-m_{\alpha}}+\sqrt{m_{\alpha}-m_{p+1}}\leq 2\sqrt{b-m_{p+1}}$ and $n^{d}\lambda\ll\exp(\log^{3}m)$, we have

[TABLE]

This finishes the proof of the lemma. ∎

Finally we combine Lemma 4.10 and Lemma 4.11 to prove Lemma 4.7.

Proof of Lemma 4.7.

Recall that $d,q\leq\log n$ and $\lambda,L\leq(\log n)^{O(\log n)}$.

Let $\alpha\in[0:r]$ and $\beta\in[0:4r+3]$ be the final parameters that satisfy Lemma 4.11 for $\Gamma_{r}=\Gamma$. We define the polynomial $f$ using $h$ from Lemma 4.10 as follows

[TABLE]

It follows from Lemma 4.10 that $f$ has degree

[TABLE]

using $r<q$ and $\beta\leq 4r+3$. Moreover, we have

[TABLE]

It follows from the definition of $\phi$ in (11) and $w_{1},\ldots,w_{d}\in[L^{2}]$ that the same degree upper bound holds for $\phi$ and

[TABLE]

To analyze $\sum_{b}f(b)\cdot\Gamma(b)$, we show that $|f(b)\cdot\Gamma(b)|\leq|\Gamma(m_{\alpha})|/(2m)$ for all $b\neq m_{\alpha}$ and thus,

[TABLE]

using $\tau_{0}\geq\|\Delta\|_{\infty}/\lambda$. For each $b>m_{\alpha}$, by Lemma 4.11 and Lemma 4.10 (and $q\leq\log n$)

[TABLE]

For each $b<m_{\alpha}$ we have from Lemma 4.11 and Lemma 4.10 that

[TABLE]

This finishes the proof of Lemma 4.7. ∎

4.4 Proof of Lemma 4.8

Let $\mathbf{X}$ and $\mathbf{Y}$ be two distributions each supported over strings from $\{0,1\}^{n}$.

Given $0\leq j_{1}<\cdots<j_{d}\leq k-1$, we use $g_{j_{1},\ldots,j_{d}}$ to denote the following $d$-variate polynomial,

[TABLE]

To see the relevance of this polynomial to the $k$-deck, we note that given any $0\leq t_{1}<\cdots<t_{d}<n$ the quantity $g_{j_{1},\dots,j_{d}}(t_{1},\dots,t_{d})$ is the number of ways to pick $k$ indices from $[0:n-1]$ such that each $t_{i}$ is the $(j_{i}+1)$th smallest index picked.

We first show that the following sum

[TABLE]

can be written as a low-weight linear combination of entries of $\mathsf{D}_{k}(\mathbf{X})$.

Lemma 4.12**.**

For any $0\leq j_{1}<\cdots<j_{d}\leq k-1$ and any $c\in\{0,1\}^{d}$, the sum (25) can be written as a linear combination of entries of $\mathsf{D}_{k}(\mathbf{X})$ in which each coefficient is either [math] or ${n\choose k}$.

Proof.

Recalling the combinatorial interpretation of $g_{j_{1},\dots,j_{d}}(t_{1},\dots,t_{d})$ given after (24), we see that if we divide the sum in (25) by ${n\choose k}$, the result is precisely the probability that $(\bm{z}_{j_{1}},\ldots,\bm{z}_{j_{d}})=c$ when we draw $\bm{x}\sim\mathbf{X}$, draw a size-$k$ subset $\mathbf{T}$ of $[0:n-1]$ uniformly at random, and then set $\bm{z}=\bm{x}_{\mathbf{T}}$. The latter probability can also be expressed using entries of $\mathsf{D}_{k}(\mathbf{X})$ as

[TABLE]

as $(\mathsf{D}_{k}(\mathbf{X}))_{z}$ is the probability of $\bm{x}_{\mathbf{T}}=z$ with $\bm{x}$ and $\mathbf{T}$ drawn as above. This finishes the proof. ∎

Next we show that, for every monomial $t_{1}^{r_{1}}\cdots t_{d}^{r_{d}}$ of degree $r_{1}+\cdots+r_{d}\leq k-d$, there exists a low-weight linear combination of polynomials $g_{j_{1},\ldots,j_{d}}$ that agrees with $t_{1}^{r_{1}}\cdots t_{d}^{r_{d}}$ over $t_{1},\ldots,t_{d}$ that satisfy $0\leq t_{1}<\cdots<t_{d}<n$.

Lemma 4.13**.**

For any nonnegative integers $r_{1},\ldots,r_{d}$ with $r_{1}+\cdots+r_{d}\leq k-d$, we have that

[TABLE]

where the coefficients $w_{j_{1},\dots,,j_{d}}$ satisfy $\sum|w_{j_{1},\dots,j_{d}}|\leq k^{O(k)}$.

Before proving Lemma 4.13, we use Lemma 4.12 and Lemma 4.13 to prove Lemma 4.8.

Proof of Lemma 4.8.

Combining Lemma 4.12 and Lemma 4.13, we have that

[TABLE]

can be written as a linear combination of entries of $\mathsf{D}_{k}(\mathbf{X})$ in which each coefficient has magnitude at most $\smash{{k^{O(k)}\cdot{n\choose k}=n^{O(k)}}}$. As a result, we have

[TABLE]

This finishes the proof of the lemma. ∎

Finally we prove Lemma 4.13. We follow a three-step approach. We say that a quasimonomial is a polynomial of the form

[TABLE]

for some nonnegative integers $\alpha_{1},\ldots,\alpha_{d}$; the degree of this quasimonomial is $\alpha_{1}+\cdots+\alpha_{d}$. And we say that a PBC (Product of Binomial Coefficients) is a polynomial of the form

[TABLE]

for some nonnegative integers $\beta_{1},\ldots,\beta_{d}$; the degree of this PBC is $\beta_{1}+\cdots+\beta_{d}$. We observe that, compared to PBCs, the polynomials $g_{j_{1},\ldots,j_{d}}$ have an extra binomial coefficient that involves $t_{d}$ at the end. The three steps of our approach are as follows:

• •

First step: Express each $d$-variable monomial $t_{1}^{r_{1}}\cdots t_{d}^{r_{d}}$ with $r_{1}+\cdots+r_{d}\leq k-d$ as a low-weight linear combination of quasimonomials of degree at most $k-d$.

• •

Second step: Express each quasimonomial of degree at most $k-d$ as a low-weight linear combination of PBCs of degree at most $k-d$.

• •

Third step: Finally, express each PBC of degree at most $k-d$ as a low-weight linear combination of polynomials $g_{j_{1},\ldots,j_{d}}$.

We elaborate on each of these three steps below. For each step, we bound the sum of magnitudes of coefficients in the linear combination.

First step. Consider the change of variables: $s_{1}=t_{1},s_{2}=t_{2}-t_{1}-1,\ldots,s_{d}=t_{d}-t_{d-1}-1$. Then

[TABLE]

Each monomial of $s_{1},\ldots,s_{d}$ on the RHS corresponds to a quasimonomial of degree at most $r_{1}+\cdots$ $+r_{d}\leq k-d$ so this gives us an expression of $t_{1}^{r_{1}}\cdots t_{d}^{r_{d}}$ as a linear combination of quasimonomials of degree at most $k-d$. Moreover, the sum of magnitudes of the coefficients is bounded by

[TABLE]

Second step. We start with a one-dimensional version of the second step.

Claim 4.14**.**

For each $r\geq 0$ and $t\geq 0$, we have

[TABLE]

Proof.

We can write $\smash{t^{r}=\sum_{\beta=0}^{r}v_{\beta}{t\choose\beta}}$ with $v\in\mathbbm{R}^{r+1}$ by changing bases in the space of polynomials in $t$. Let $P$ be the $\smash{(r+1)\times(r+1)}$ Pascal matrix with $\smash{P_{i,j}={i\choose j}}$, and define $u\in\mathbbm{R}^{r+1}$ by $u_{i}=i^{r}$. Then $\smash{u=Pv}$ so $\smash{v=P^{-1}u}$. By Theorem 2 of [BP92], we have

[TABLE]

as desired. ∎

We use Claim 4.14 $d$ times to re-express each of $t_{1}^{\alpha_{1}}$, $(t_{2}-t_{1}-1)^{\alpha_{2}},\dots,(t_{d}-t_{d-1}-1)^{\alpha_{d}}$ as a linear combination of binomial coefficients. As a result, when $0\leq t_{1}<\cdots<t_{d}<n$ we have

[TABLE]

for coefficients $c_{\beta_{1},\dots,\beta_{d}}$ that we will proceed to bound. Note that the final sum is over $0\leq\beta_{i}\leq\alpha_{i}$, so this is a linear combination of PBCs of degree at most $\alpha_{1}+\cdots+\alpha_{d}\leq k-d$.

Now we bound the sum of magnitudes of coefficients. For $0\leq\beta\leq\alpha$ we have

[TABLE]

which implies (using $\alpha_{1}+\ldots+\alpha_{d}\leq k-d\leq k$)

[TABLE]

Third step: The next claim gives an expression of a PBC as a linear combination of $g_{j_{1},\ldots,j_{d}}$’s.

Claim 4.15** (Third step: $d$-variable combinatorial identity).**

Given any $0\leq t_{1}<\cdots<t_{d}<n$ and any nonnegative integers $\beta_{1},\ldots,\beta_{d}$ with $\beta_{1}+\cdots+\beta_{d}\leq k-d$, we have

[TABLE]

Proof.

Assume that $0\leq t_{1}<\cdots<t_{d}<n$. We first consider the following combinatorial experiment with $n$ balls numbered $[0:n-1]$: (1) Mark $k$ of the $n$ balls, including balls $t_{1},\dots,t_{d}$; (2) Color red $\beta_{1}+\cdots+\beta_{d}+d$ of the $k$ marked balls, including balls $t_{1},\dots,t_{d}$, in such a way that for each $i\in[d]$, the $(\beta_{1}+\cdots+\beta_{i}+i)$-th red one is $t_{i}$ (i.e., there are $\beta_{1}$ red balls before $t_{1}$, $\beta_{2}$ red balls between $t_{1}$ and $t_{2}$, …, and $\beta_{d}$ red balls between $t_{d-1}$ and $t_{d}$). Below we count the total number of outcomes of this experiment (including which balls are marked and which balls are colored red) in two different ways to obtain the desired identity.

In the first way, we note that at the end of this experiment there are $\beta_{1}$ balls that are marked and red within the $t_{1}$ balls $\{0,\dots,t_{1}-1\}$ (and also $t_{1}$ is marked and red), and for each $i\in[2:d]$ there are $\beta_{i}$ balls that are marked and red within the $t_{i}-t_{i-1}-1$ balls $\{t_{i-1}+1,\dots,t_{i}-1\}$ (and also $t_{i}$ is marked and red); and there are $k-\beta_{1}-\cdots-\beta_{d}-d$ other balls that are marked within the $n-\beta_{1}-\cdots-\beta_{d}-d$ other balls. So the total number of outcomes of the experiment is precisely

[TABLE]

We can also count the number of outcomes of the experiment in a different way, by viewing the experiment as being carried out as follows: (a) For some numbers $0\leq j_{1}<\cdots<j_{d}<k$, mark $k$ balls such that for each $i\in[d]$, the $(j_{i}+1)$-th marked ball is $t_{i}$; note that as mentioned earlier, $g_{j_{1},\dots,j_{d}}(t_{1},\dots,t_{d})$ is precisely the number of ways to do this. (b) Color red $\beta_{1}$ of the $j_{1}$ marked balls before ball $t_{1}$ (and also color red ball $t_{1}$; there are ${j_{1}\choose\beta_{1}}$ ways to do this), and for each $i\in[2:d]$, color red $\beta_{i}$ of the $j_{i}-j_{i-1}-1$ marked balls that lie in $\{t_{i-1}+1,\dots,t_{i}-1\}$ (and also color red $t_{i}$; there are $\smash{j_{i}-j_{i-1}-1\choose\beta_{i}}$ ways to do this). From this perspective, the total number of outcomes is

[TABLE]

and we have established the identity as desired. ∎

Observe that when $\beta_{1}+\cdots+\beta_{d}\leq k-d$, we have

[TABLE]

which implies that the sum of magnitudes of coefficients in the linear combination is

[TABLE]

We can now combine the three steps to prove Lemma 4.13.

Proof of Lemma 4.13.

We express $t_{1}^{r_{1}}\cdots t_{d}^{r_{d}}$ as a linear combination of polynomials $g_{j_{1},\ldots,j_{d}}$ via the three steps as described above, with coefficients $w_{j_{1},\ldots,j_{d}}$. The sum of magnitudes of coefficients in the linear combination used in the First, Second, and Third steps are bounded from above using inequalities (26), (28), and (29) respectively. These bounds give us

[TABLE]

as desired. This finishes the proof of the lemma. ∎

5 Lower bounds for distributions supported on at most ${2\ell}$ strings

Our main result in this section is Theorem 6, given below, which establishes a lower bound on the sample complexity of population recovery under the deletion channel which is exponential in the population size for a wide range of population sizes:

Theorem 6**.**

Fix any constant deletion probability $\delta\in(0,1)$. Suppose that $A$ is an algorithm which, when run on i.i.d. samples drawn from a distribution $\mathrm{Del}_{\delta}(\mathbf{X})$ with $|\mathrm{supp}(\mathbf{X})|\leq 2{\ell}$, outputs a hypothesis $\tilde{\mathbf{X}}$ which satisfies $d_{\mathrm{TV}}(\mathbf{X},\tilde{\mathbf{X}})\leq 0.49$ with probability at least $0.51.$ Then $A$ must use

[TABLE]

many samples.

If the population size upper bound $2\ell$ is a constant this gives a lower bound of $\Omega(n^{(\ell+1)/2})$ samples, and for any $\ell<n^{0.499}$ this gives a lower bound of $n^{\Omega(\ell)}.$

For the rest of this section fix $\delta\in(0,1)$ and let $\rho$ denote $1-\delta$. The high-level idea of the proof is as follows: We show that there exist two distributions $\mathbf{X},\mathbf{Y}$ over $\{0,1\}^{n}$ which have disjoint supports, each of size at most ${2\ell}$, but satisfy

[TABLE]

which clearly implies Theorem 6.

For simplicity throughout this section we assume that $n$ is odd, and we write $m$ to denote $(n-1)/2.$ The following notation will be useful: For $0\leq i\leq 2\ell$ we write $e_{m+i}$ to denote the string $0^{m+i}10^{m-i}.$ The two distributions $\mathbf{X}$ and $\mathbf{Y}$ that we consider will be supported on disjoint subsets of $\{e_{m+i}\}_{i\in[0:2\ell]}$ (and hence each distribution has support size at most $2\ell+1$, but in our proofs neither will have full support so their support size will be at most $2\ell$).

Notation and setup. For notational convenience, let $\mathrm{B}(r)$ denote the binomial distribution $\mathrm{Bin}(r,{\rho})$.

Let $S$ be a set of indices, $\pi_{S}$ be a distribution over $S$, and $\{\mathbf{V}_{i}\}_{i\in S}$ be a set of random variables indexed by $S$. We write $\mathrm{Mix}(\pi_{S};\{\mathbf{V}_{i}\}_{i\in S})$ to denote the mixture over $\{\mathbf{V}_{i}\}_{i\in S}$ with each $\mathbf{V}_{i}$ weighted by $\pi_{S}(i)$.

For conciseness we write $\mathbf{Z}_{n}$ to denote a random variable which is distributed according to the binomial distribution $\mathrm{B}(n).$ We recall the following convenient expression for the falling moments of the binomial distribution: for any $t=0,1,\ldots$, we have

[TABLE]

For completeness we include the derivation below:

[TABLE]

The key lemmas. The first main lemma makes precise the moment-matching property of $\pi_{S}$ and $\pi_{T}$ that we require:

Lemma 5.1** (Matching moments of mixtures of disjointly supported binomial distributions).**

Let $\ell\leq O(\sqrt{n})$.141414Note that if $\ell=\omega(\sqrt{n})$ then Theorem 6 holds trivially, so this assumption is without loss of generality. There are two disjoint subsets $S,T\subset[0:2\ell]$ and two distributions $\pi_{S},\pi_{T}$ supported on $\{e_{m+i}\}_{i\in S}$ and $\{e_{m+j}\}_{j\in T}$ respectively with the following property (which we call the “matching moment property”):

Let $\tilde{\mathbf{D}}_{S}$ be a random variable whose distribution is the mixture of $\{\mathbf{Z}_{m+i}\}_{i\in S}$ in which distribution $\mathbf{Z}_{m+i}$ has mixing weight $\pi_{S}(e_{m+i})$, and likewise $\tilde{\mathbf{D}}_{T}$ be a random variable whose distribution is the mixture of $\{\mathbf{Z}_{m+j}\}_{j\in T}$ in which distribution $\mathbf{Z}_{m+j}$ has mixing weight $\pi_{T}(e_{m+j})$. Then the first ${\ell}$ moments of $\tilde{\mathbf{D}}_{S}$ and $\tilde{\mathbf{D}}_{T}$ match each other, i.e. for all $t\in[\ell]$, we have

[TABLE]

The second main lemma (statement given in Lemma 5.3 below) gives the desired upper bound on total variation distance. To prove Theorem 6 it suffices to prove Lemmas 5.1 and 5.3.

5.1 Proof of Lemma 5.1

Proof.

The proof is by a linear algebraic argument. Let $r=m+\ell$. Consider the mixtures

[TABLE]

and

[TABLE]

where all $a_{i},b_{i}\in[0,1]$ and $\sum_{i=-\ell}^{\ell}a_{{|i|}}=\sum_{j=-\ell}^{\ell}b_{{|j|}}=1.$ Let $c_{i}=a_{i}-b_{i}$ for $0\leq i\leq\ell$.

We will prove the existence of a non-trivial solution $a_{i},b_{i}$ (i.e., such that $a_{i}\neq b_{i}$ for some $i$) such that the following system holds:

[TABLE]

Observe that this is the same as requiring that $\operatorname{{\bf E}}[\tilde{\mathbf{D}}_{S}^{t}]=\operatorname{{\bf E}}[\tilde{\mathbf{D}}_{T}^{t}]$ for $t\leq\ell$. In (33), we will be viewing $\operatorname{{\bf E}}[Q(\tilde{\mathbf{D}}_{S})]$ and $\operatorname{{\bf E}}[Q(\tilde{\mathbf{D}}_{T})]$ (for polynomials $Q$) as polynomials in $n$. We want the equations in (33) to hold as polynomial equalities.

Note that for $t\geq 0$, by (31) we can rewrite the condition $\operatorname{{\bf E}}[\tilde{\mathbf{D}}_{S}(\mathbf{D}_{S}-1)\cdots(\tilde{\mathbf{D}}_{S}-t)]=\operatorname{{\bf E}}[\tilde{\mathbf{D}}_{T}(\tilde{\mathbf{D}}_{T}-1)\cdots(\tilde{\mathbf{D}}_{T}-t)]$ as the condition

[TABLE]

viewing both sides as formal polynomials in $r$. Since $P_{t}(x)$ has degree $t+1$ in $x$, for $0\leq\ell\leq t+1$ the coefficient of $r^{\ell}$ in the polynomial on the LHS of (34) is zero, and consequently we get a system of $t+2$ homogeneous linear equations in $c_{0},c_{1},\ldots,c_{\ell}$.

Naively, it seems that (33) gives us $2+3+\cdots+\ell+1={{\ell+2}\choose 2}-1$ many homogeneous linear equations, which is far more than the $\ell+1$ variables $c_{0},\dots,c_{\ell}$ that are in play. At this point it is unclear that (33) necessarily has a nonzero solution in the $c_{i}$’s. We will show that (33) is actually comprised of at most $\ell$ equations and hence it must have a nonzero solution.

Thus, to prove the existence of a non-trivial solution to (33) phrased in terms of $\tilde{\mathbf{D}}_{S}$ and $\tilde{\mathbf{D}}_{T}$, it suffices to prove the existence of a non-trivial solution to (33) phrased in terms of polynomial equalities.

To do this, we observe that equation (34) is also true when we replace $r$ by $r+1$ and get the condition

[TABLE]

as a polynomial in $r$. (Note that the initial assumption $\ell\leq\Omega(n)$ still holds if we increase $n$.)

Observe that

[TABLE]

so if we subtract (34) from (35) and divide by ${\rho}(t+1)$, then we get the condition

[TABLE]

as a polynomial in $r$. Since this is true for all $t$, then we have derived the condition $\operatorname{{\bf E}}[\tilde{\mathbf{D}}_{S}(\tilde{\mathbf{D}}_{S}-1)\cdots(\tilde{\mathbf{D}}_{S}-t+1)]=0$. It follows by induction that all of (33) follows from the condition $\operatorname{{\bf E}}[\tilde{\mathbf{D}}_{S}(\tilde{\mathbf{D}}_{S}-1)\cdots(\tilde{\mathbf{D}}_{S}-\ell+1)]=0$.

Thus we have reduced (33) to a system of $\ell+1$ homogeneous linear equations over $\ell+1$ variables, but the first equation (which comes from observing that the coefficient of $r^{\ell}$ in the LHS of (34) is 0) will be

[TABLE]

and a second equation (which comes from observing that the coefficient of $r^{\ell-1}$ in the LHS of (34) is 0) will be

[TABLE]

because the coefficient of $r^{\ell-1}$ in $P_{\ell}(r)$ is $-(\ell/2)(\ell-1)$. This is just equation (36) times $-(\ell/2)(\ell-1)$. So there are actually at most $\ell$ distinct equations in $\ell+1$ variables, and hence there is (at least) a line of non-trivial solutions in the $c_{i}$’s.

Given a satisfying assignment to the $c_{i}$’s, for each $i$ with $c_{i}=0$ we set $a_{i}=b_{i}=0$. If $c_{i}>0$, then we set $a_{i}=c_{i}$ and $b_{i}=0$. If $c_{i}<0$, then we set $a_{i}=0$ and $b_{i}=-c_{i}$. Note that

[TABLE]

and that by homogeneity, we can scale all the $c_{i}$’s by any multiplicative constant and still get a valid solution to (33). We scale the $c_{i}$’s so that $2a_{\ell}+2a_{\ell-1}+\cdots+2a_{1}+a_{0}=1$. The above equation implies that $2b_{\ell}+2b_{\ell-1}+\cdots+2b_{1}+b_{0}=1$ as well.

This results in the coefficients $a_{i}$ and $b_{i}$ satisfying (33) and $\tilde{\mathbf{D}}_{S}$ and $\tilde{\mathbf{D}}_{T}$ being valid distributions that are disjointly supported. Since the $c_{i}$’s were non-trivial, then at least one coefficient $c_{i}$ is non-zero and by equation (36), there exist coefficients $c_{j}$ and $c_{k}$ of opposite sign. Thus, both $\tilde{\mathbf{D}}_{S}$ and $\tilde{\mathbf{D}}_{T}$ have support sizes at most $2\ell$.

We take $\pi_{S}(e_{m+t})=a_{|t-\ell|}$ and $\pi_{T}(e_{m+t})=b_{|t-\ell|}$ to conclude the proof. ∎

We will use the following corollary of Lemma 5.1:

Corollary 5.2**.**

Let $S,T,\pi_{S},\pi_{T}$ be as in Lemma 5.1. Then for any polynomial $p$ of degree at most $\ell$, we have

[TABLE]

Proof.

Equation (32) can be rewritten as

[TABLE]

which holds for all $t\leq\ell$. This is equivalent to having equal falling moments, i.e. for all $t\in[\ell],$

[TABLE]

Indeed, for a random variable $\mathbf{Z}$, $\operatorname{{\bf E}}[P_{t-1}(\mathbf{Z})]$ can be written as a linear combination of $1,\operatorname{{\bf E}}[\mathbf{Z}],\operatorname{{\bf E}}[\mathbf{Z}^{2}],$ $\ldots,$ $\operatorname{{\bf E}}[\mathbf{Z}^{t}]$ and since $1,P_{0}(\mathbf{Z}),P_{1}(\mathbf{Z}),\ldots,P_{\ell-1}(\mathbf{Z})$ form a set of $\ell$ polynomials in $\mathbf{Z}$ with degrees $0,1,2,\ldots,\ell$, then they form a basis for polynomials in $\mathbf{Z}$ with degree at most $\ell$.

By (31), this is in turn equivalent to having, for all $t\in[\ell],$

[TABLE]

which is in turn equivalent to (37) by the reasoning in the above paragraph. ∎

5.2 Total Variation Distance Upper Bound

We state Lemma 5.3 below. Informally, it says that if $\pi_{S},\pi_{T}$ have the matching moment property, then the variation distance between two corresponding mixtures of two-dimensional vector-valued random variables is small. (In the following, the notation $(\mathrm{B}(a),\mathrm{B}(b))$ stands for a vector-valued random variable in which the two coordinates are independently drawn from $\mathrm{B}(a)$ and $\mathrm{B}(b)$ respectively.)

Lemma 5.3**.**

Let $\mathbf{X},\mathbf{Y}$ be two distributions with disjoint supports $\{e_{m+i}\}_{i\in S}$ and $\{e_{m+j}\}_{j\in T}$ respectively, where $S\cup T\subset[0:2\ell]$, with the matching moment property from Lemma 5.1 above. Then

[TABLE]

Setup and useful results. Our proof of Lemma 5.3 is based on “moment-matching” results for Poisson binomial distributions which were proved by Roos [Roo00] and subsequently used by Daskalakis and Papadimitriou [DP15]. Our approach is similar to the approach used in [DP15]. To state these results, recall that a Poisson binomial distribution (PBD) is a sum $\mathbf{U}=\mathbf{A}_{1}+\cdots+\mathbf{A}_{n}$ of independent Bernoulli random variables (so each $\mathbf{A}_{i}$ is a random variable taking value 1 with some probability $p_{i}\in[0,1]$ and taking value 0 with probability $1-p_{i}$).

In [DP15], it is shown that if two PBDs satisfy some mild technical condition and have matching first $\ell$ moments, then they have total variation distance at most $2^{-\Omega(\ell)}$. We show that two mixtures of pairs of suitable binomially distributed variables that have matching first $\ell$ moments will have total variation distance at most $n^{-\Omega(\ell)}$.

We recall Theorem 1 of [DP15], which gives a “Krawtchouk expansion” for any Poisson binomial distribution. This provides an expression for the exact probability that the Poisson binomial distribution puts on any outcome in its support. (We state the theorem for PBDs which are a sum of $n^{\prime}$ many random variables, as when we apply it later it will be for such PBDs where $n^{\prime}=m+\ell=(n-1)/2+\ell.$)

Theorem 7** (Theorem 1 of [Roo00], see also Theorem 7 of [DP15]).**

Let $\mathbf{U}=\mathbf{A}_{1}+\cdots+\mathbf{A}_{n^{\prime}}$ be a Poisson binomial distribution in which each $\mathbf{A}_{i}$ takes value 1 with probability $p_{i}\in[0,1]$. Then for all $r\in[n^{\prime}]$ and all $p\in[0,1]$, we have

[TABLE]

where

• •

$\alpha_{0}(p_{1},\dots,p_{n^{\prime}};p)=1$* and for $t\in[0:n^{\prime}]$,*

[TABLE]

• •

and for all $t\in[0:n^{\prime}]$,

[TABLE]

where in the last expression $B_{n^{\prime},p}(r)$ denotes the value ${n^{\prime}\choose r}p^{r}(1-p)^{n^{\prime}-r}$, the probability that the binomial distribution $\mathrm{Bin}(n^{\prime},p)$ puts on the outcome $r$, viewed as a function of $p$.

We highlight the fact that $\Delta^{t}B_{n^{\prime},p}(r)$ has no dependence on the parameters $p_{1},\dots,p_{n^{\prime}}$; this will be important for us later.

The following result, deduced from [Roo00], is very useful in analyzing (39). It bounds each of the $n^{\prime}+1$ summands in (39) which add up to $\operatorname{{\bf Pr}}[\mathbf{U}=r]$.

Theorem 8**.**

Let $(p_{1},\dots,p_{n^{\prime}})\in[0,1]^{n^{\prime}}$, $p\in[0,1]$, and $\alpha_{t}(\cdot,\cdot)$ be as in the statement of Theorem 7. Define

[TABLE]

For $t\in[n^{\prime}]$,

[TABLE]

where $\|\Delta^{t}B_{n^{\prime}-t,p}(\cdot)\|_{1}$ denotes the 1-norm of $\Delta^{t}B_{n^{\prime}-t,p}(\cdot)$ viewed as an $(n^{\prime}+1)$-dimensional vector, i.e. $\|\Delta^{t}B_{n^{\prime}-t,p}(\cdot)\|_{1}:=\sum_{r=0}^{n^{\prime}}\left|\Delta^{t}B_{n^{\prime}-t,p}(r)\right|$.

Proof.

Inequality (30) in [Roo00] gives

[TABLE]

for $t\in[n^{\prime}]$.

Inequality (38) in [Roo00] gives

[TABLE]

for $t\in[n^{\prime}]$.

By multiplying the above two inequalities together we get the desired result because $t\leq n^{\prime}$. ∎

For conciseness we now let $\mathbf{D}_{S}$ denote $\mathrm{Mix}(\pi_{S};((\mathrm{Bin}(m+i,\rho),\mathrm{Bin}(m-i,\rho)))_{i\in S})$ where in each component two-dimensional distribution the two distributions $\mathrm{Bin}(m+i,\rho)$ and $\mathrm{Bin}(m-i,\rho)$ are independent, and similarly we let $\mathbf{D}_{T}$ denote $\mathrm{Mix}(\pi_{T};((\mathrm{Bin}(m+j,\rho),\mathrm{Bin}(m-j,\rho)))_{j\in T})$. In the rest of the proof we will argue that

[TABLE]

This establishes the claimed upper bound on $d_{\mathrm{TV}}(\mathrm{Del}_{\delta}(\mathbf{X}),\mathrm{Del}_{\delta}(\mathbf{Y}))$ given in (38). To see this, observe that for any outcome in $\mathrm{supp}(\mathbf{X})$ or $\mathrm{supp}(\mathbf{Y})$, with probability $\delta$ the one 1-coordinate is deleted under $\mathrm{Del}_{\delta}$ (in which case the distributions resulting from $\mathrm{Del}_{\delta}(\mathbf{X})$ and $\mathrm{Del}_{\delta}(\mathbf{Y})$ are identical), and that with the remaining $1-\delta$ probability (when the one 1-coordinate is not deleted) there is an exact correspondence between $\mathrm{Del}_{\delta}(\mathbf{X})$ and $\mathbf{D}_{S}$ and between $\mathrm{Del}_{\delta}(\mathbf{Y})$ and $\mathbf{D}_{T}$.

For an index $c\leq n^{\prime}$, let $v^{(c)}$ denote the $n^{\prime}$-dimensional real vector whose first $c$ values are $\rho$ and whose remaining values are [math].

For $t,t^{\prime}\in[0:n^{\prime}]$ we define

[TABLE]

The following lemma is crucial for us. Recall that $n^{\prime}=m+\ell.$

Lemma 5.4**.**

Let $\pi_{S},\pi_{T}$ be as in the statement of Lemma 5.3. Then for any $p\in[0,1]$, the values $C_{t,t^{\prime}}(p)$ and $D_{t,t^{\prime}}(p)$ are identical for $t,t^{\prime}\geq 0$ and $t+t^{\prime}\leq\ell$.

Proof.

Let $p$ be any value in $[0,1]$. If $t+t^{\prime}=0$, then $t=t^{\prime}=0$. Recalling that $\alpha_{0}(\cdot,\cdot)\equiv 1$ we have that

[TABLE]

as desired.

For $t+t^{\prime}\geq 1,$ we observe that $\alpha_{t}(v^{(m+i)};p)\cdot\alpha_{t^{\prime}}(v^{(m-i)};p)$ is composed of summands of the form $({\rho}-p)^{c+c^{\prime}}(-p)^{t+t^{\prime}-c-c^{\prime}}$ for $c\in[0,t],c^{\prime}\in[0,t^{\prime}]$.

In particular, we have

[TABLE]

in which each $\pi_{S}(e_{m+i})$ is multiplied by a polynomial in $m$ of degree at most $t+t^{\prime}\leq\ell$.

Similarly, we have

[TABLE]

and by Corollary 5.2, we see that $C_{t,t^{\prime}}(p)=D_{t,t^{\prime}}(p)$. ∎

Now we proceed to prove Lemma 5.3. Our argument is similar to the proof of Theorem 3 in [DP15].

Let $p\in[0,1]$ and $r,s\in[0:n^{\prime}]$. We have

[TABLE]

where the two lines are by Theorem 7 and Lemma 5.4 respectively. As a result, for any $p\in[0,1]$ we have

[TABLE]

and expanding out definitions gives

[TABLE]

By Theorem 8,

[TABLE]

where the second inequality can be deduced from the AM-GM inequality.

Fix any $i\in S,j\in T$. Let $p={\rho}$ in Theorem 8. Since $\rho$ is a constant in $(0,1)$ we get that

[TABLE]

and similarly

[TABLE]

because $\ell\leq O(\sqrt{n})$ (and we may assume that $\ell\leq O(\sqrt{n})$ since otherwise the total variation distance bound claimed in the lemma is trivial).

By choosing sufficiently large $n$ and appropriate constants, we can upper bound the RHS by some $\theta<1/2$. This gives

[TABLE]

where the second inequality comes from the fact that there are $i+1$ pairs of non-negative integers $t,t^{\prime}$ that sum to $i$, and the third inequality comes from the fact that $i^{\frac{3}{2}}\leq(\ell+1)^{\frac{-1}{2}}i^{2}$ when $i>\ell$.

Observe that

[TABLE]

for $0<x<1$, so

[TABLE]

for $0<x<1/2$. This means

[TABLE]

giving (42) as desired and concluding the proof of Lemma 5.3.

Appendix A Proof of Lemma 4.10

A.1 Chebyshev polynomials

Let $T_{r}(x)$ denote the $r$th Chebyshev polynomial of the first kind. Then $T_{r}$ has degree $r$ and satisfies the following property:

Property 9**.**

$T_{r}(1)=1$* and $|T_{r}(x)|\leq 1$ for all $|x|\leq 1$. If $x>1$ then $T_{r}(x)>1$.*

We will need an upper bound for $T_{r}(x)$ over $x\in[1,2]$. For this purpose we recall the following explicit form of $T_{r}(x)$ for $|x|\geq 1$:

[TABLE]

Property 10**.**

For $a\in[0,1]$, we have $T_{r}(1+a)\leq e^{3r\sqrt{a}}$.

Proof.

Using (43) we have

[TABLE]

where we used $a^{2}\leq a\leq\sqrt{a}$ when $a\in[0,1]$ and $1+x\leq e^{x}$. ∎

The next property follows from the recurrence relation

[TABLE]

with initial conditions $T_{0}(x)=0$ and $T_{1}(x)=1.$

Property 11**.**

*For all $r\geq 0$, we have $\|T_{r}\|_{1}\leq 3^{r}.$ *

Following [BEK99], we write $g_{r}$ to denote the following polynomial of degree $r$:

[TABLE]

We need the following properties of the polynomial $g_{r}$. Items 1, 2 and 3 of Property 12 follow directly from Properties 9, 10 and 11, respectively. For item 4 we have

[TABLE]

for all $0<y\leq\pi$. This implies that for all $x\in[-1,1)$, we have

[TABLE]

Property 12**.**

The polynomial $g_{r}$ satisfies the following properties.

$g_{r}(1)=1$* and $|g_{r}(x)|\leq 1$ for all $|x|\leq 1$;* 2. 2.

$1\leq g_{r}(1+a)\leq e^{3r\sqrt{a}}$* for all $a\in[0,1]$;* 3. 3.

$\|g_{r}\|_{1}\leq 3^{r}$; and 4. 4.

$|g_{r}(x)|\leq\frac{1}{r\sqrt{2(1-x)}}$* for all $x\in[-1,1)$.*

A.2 Proof of Lemma 4.10

Recall the statement of Lemma 4.10:

Lemma 4.10, restated. * There is a univariate polynomial $h$ with the following properties:*

$h$* has degree $O(\sqrt{m}\log m)$.* 2. 2.

$h(0)=1$* and for each $b\in[m]$,*

[TABLE] 3. 3.

$h$* satisfies $\|h\|_{1}\leq\exp(O(\sqrt{m}\log m))$.*

Proof.

Recall the polynomial $g_{r}$ in Section A.1. We use it to define a degree-$r$ polynomial $\psi_{r}$:

[TABLE]

Properties of $g_{r}$ directly imply the following properties of $\psi_{r}$:

$\psi_{r}(0)=1$; 2. 2.

For each $b\in[m]$, we have

[TABLE] 3. 3.

Finally, $\psi_{r}$ satisfies

[TABLE]

Let $\tilde{m}=4^{\beta}$ be the smallest power of $4$ with $\tilde{m}\geq m$. We use $\psi_{r}$ to define our $h$ as follows:

[TABLE]

First we have $h(0)=1$ and the degree of $h$ is at most

[TABLE]

Next, given $b\in[m]$, let $i\in[\beta]$ be an integer such that $4^{i-1}\leq b\leq 4^{i}$. Then (using $\tilde{m}\geq m$)

[TABLE]

Using $|\psi_{r}(b)|\leq 1$ for all $r$, we have that

[TABLE]

On the other hand, we have for each $b\in[m]$ (using $\tilde{m}\leq 4m$ and that $m$ is asymptotically large),

[TABLE]

Finally, the sum of magnitudes of coefficients of $h$ is at most

[TABLE]

This finishes the proof of the lemma. ∎