A local limit theorem for Quicksort key comparisons via multi-round smoothing

TL;DR

This paper proves a full local limit theorem for the distribution of key comparisons in Quicksort using a novel multi-round smoothing technique, building on prior global and convergence results.

Contribution

It introduces a multi-round smoothing method to establish the local limit theorem for Quicksort key comparisons, advancing the understanding of its distributional properties.

Findings

Proves the full local limit theorem for Quicksort comparisons

Develops a new multi-round smoothing technique

Complements previous global distribution results

Abstract

As proved by R\'egnier and R\"osler, the number of key comparisons required by the randomized sorting algorithm QuickSort to sort a list of $n$ distinct items (keys) satisfies a global distributional limit theorem. Fill and Janson proved results about the limiting distribution and the rate of convergence, and used these to prove a result part way towards a corresponding local limit theorem. In this paper we use a multi-round smoothing technique to prove the full local limit theorem.

Equations

$$\mathbb{P}(Q_n=x)=n^{-1}f((x-q_n)/n)+O(n^{-1-\epsilon })$$

$$(X_n-{\mu }_n)/{\sigma }_n\stackrel{\mathrm{d}}{\to }X,$$

$$\mathbb{P}(X_n=x_n)={\sigma }_n^{-1}f((x_n-{\mu }_n)/{\sigma }_n)+o({\sigma }_n^{-1}).$$

$$\mathbb{P}(X_n=x_n)=\mathbb{P}(X_n=x_n^{\prime })+o({\sigma }_n^{-1}).$$

$$\mathbb{P}(Q_n\in I_1)=\mathbb{P}(Q_n\in I_2)+o(\ell /n).$$

$$-{\alpha }_i⩽Z_i-Z_{i-1}⩽{\beta }_i.$$

$$\mathbb{P}(Z_n-\mu ⩾a)⩽\exp \left\{-2a^2/\sum _{i=1}^n({\alpha }_i+{\beta }_i{)}^2\right\},$$

$$\sup_{x}\bigl{|}{\mathbb{P}}(S\leqslant x)-\Phi((x-\mu)/\sigma)\bigr{|}\leqslant A\rho/\sigma^{3},$$

$$\mathbb{P}\left(X_{n,r}⩽\frac{n}{3r}\right)⩽e^{-n/(400r)}.$$

$$t_0=\lceil 20n/r\rceil .$$

$$\mathbb{P}(T⩾t_0)⩽\mathbb{P}(N⩾t_0/2).$$

$$2\frac{(r/10+1)}{\ell }⩽\frac{3r/10}{\ell }<\frac{3}{10},$$

$${\mathbb{P}}\bigl{(}T\geqslant t_{0}\bigr{)}\leqslant{\mathbb{P}}\bigl{(}N\geqslant t_{0}/2\bigr{)}\leqslant{\mathbb{P}}\bigl{(}\mathrm{Bin}(t_{0},3/10)\geqslant t_{0}/2\bigr{)}\leqslant e^{-2t_{0}/25},$$

$$X_0=\cdots =X_{(r/2)-1}=0,$$

$$X_{r/2}=\cdots =X_r=1,$$

$$X_n\stackrel{\mathcal{L}}{=}{X}_{U_n-1}+X_{n-U_n}^{\ast },n⩾r+1,$$

$${\xi }_n:=E{X}_n.$$

$${\xi }_0=\cdots ={\xi }_{(r/2)-1}=0,$$

$${\xi }_{r/2}=\cdots ={\xi }_r=1,$$

$${\xi }_n=\frac{n+1}{r+1},n⩾r+1.$$

$$|{\xi }_n-{\tilde{\xi }}_n|⩽\frac{r/2}{r+1}<\frac{1}{2},$$

$$-1<{\xi }_{k-1}+{\xi }_{n-k}-{\xi }_n<1$$

$$M_t=E[X_n\mid {\mathcal{F}}_t],$$

$$-1<M_t-M_{t-1}<1$$

$${\xi }_n=\frac{n+1}{r+1}⩾\frac{n}{r+1}⩾\frac{2n}{3r},$$

$$\mathbb{P}(E)⩽\mathbb{P}\left(T>t_0\right)+\mathbb{P}\left(M_{t_0}-{\xi }_n⩽-\frac{n}{3r}\right).$$

$$\mathbb{P}(E)$$

$$\mathbb{P}(Q_n^{\ast }\in [-2,2])⩾2c_1$$

$$\mathrm{Var}(Q_n^{\ast }\mid Q_n^{\ast }\in [-2,2])⩾c_1.$$

$${\mathbb{P}}\bigl{(}X\leqslant\operatorname{\mathbb{E}{}}X+x\sqrt{\operatorname{Var}X}\bigr{)}=\Phi(x)+O(1/\sqrt{s}),$$

$$\mathbb{P}(X\in [t,t+\ell ])⩽\frac{C\ell }{r\sqrt{s}}{e}^{-c{t}^2/(r^{2}s)}.$$

Full text

1 Introduction

QuickSort, a basic sorting algorithm, may be described as follows. The input is a list, of length $n\geqslant 0$, of distinct real numbers (say). If $n=0$ or $n=1$, do nothing (the list is already sorted). Otherwise, pick an element of the list uniformly at random to use as the pivot, and compare each other element with the pivot. Recursively sort the two resulting sublists, and combine them in the obvious way, with the pivot in the middle. (Equivalently, one can sort the initial list randomly, and always use the first element in each (sub)list as the pivot.) The recursive calls to the algorithm lead to a tree, the execution tree, with one node for each call. Each node either has no children (if the corresponding list had length [math] or $1$) or two children. The main quantity we study here is the random variable $Q_{n}$, the total number of comparisons used in sorting a list of $n$ distinct items.

Régnier [11] and Rösler [13] each established, using different methods, a distributional limit theorem for $Q_{n}$, proving that $(Q_{n}-\operatorname{\mathbb{E}{}}Q_{n})/n\overset{\mathrm{d}}{\to}Q$ as $n\to\infty$, where $Q$ has a certain distribution that can be characterized in a variety of ways—to name one, as the unique fixed point of a certain distributional identity. Using that distributional identity, Fill and Janson [5] showed (among stronger results) that the distribution of $Q$ has a continuous and strictly positive density $f$ on ${\mathbb{R}}$.

Fill and Janson [6] proved bounds on the rate of convergence in various metrics, including the Kolmogorov–Smirnov distance (i.e., sup-norm distance for distribution functions). Using this and their results about $f$ from [5], they proved a ‘semi-local’ limit theorem for $Q_{n}$; see their Theorem 6.1, which is reproduced in large part as Theorem 14 below. They posed the question [6, Open Problem 6.2] of whether the corresponding local limit theorem (LLT) holds. Here we show that the answer is yes, using a multi-round smoothing technique developed in an initial draft of [2], but not used in the final version of that paper. This method may well be applicable to other distributions in which one can find ‘smooth parts’ on various different scales, including other distributions obeying recurrences of a type similar to that obeyed by $Q_{n}$. Taking the ‘semi-local’ limit theorem of [6] as a starting point, in this paper we shall prove the following LLT for $Q_{n}$, together with an explicit (but almost certainly not sharp) rate of convergence.

Theorem 1**.**

Defining $Q_{n}$ and $Q$ as above, and setting $q_{n}:=\operatorname{\mathbb{E}{}}Q_{n}$, there exists a constant $\varepsilon>0$ such that the following holds. We have

[TABLE]

uniformly in integers $x$, where $f$ is the continuous probability density function of $Q$.

In fact, our proof of Theorem 1 gives a bound of the form $O(n^{-19/18}\log n)$ on the error probability in (1).

The basic idea used in our proof, that of strengthening a distributional (often normal) limit theorem to a local one by smoothing, is by now quite old. Suppose that $X_{n}$ takes integer values, and that we know that

[TABLE]

for some nice distribution $X$ (say with continuous, strictly positive density $f$ on ${\mathbb{R}}$). By the corresponding LLT we mean the statement that whenever $x_{n}$ is a sequence of deterministic values with $x_{n}=\mu_{n}+O(\sigma_{n})$ then

[TABLE]

It is not hard to see that to deduce (3) from (2), it suffices to show that ‘nearby’ values have similar probabilities, i.e., that if $x_{n},x_{n}^{\prime}=\mu_{n}+O(\sigma_{n})$ and $x_{n}-x_{n}^{\prime}=o(\sigma_{n})$, then

[TABLE]

In turn, to prove (4) we might (as in MacDonald [8]) try to find a ‘smooth part’ within the distribution of $X_{n}$. More precisely, we might try to write $X_{n}=A_{n}+B_{n}$ where, for some $\sigma$-algebra $\mathcal{F}_{n}$, we have that $A_{n}$ is $\mathcal{F}_{n}$-measurable and the conditional distribution of $B_{n}$ given $\mathcal{F}_{n}$ obeys (or nearly always obeys) a relation corresponding to (4). Then it follows easily (by first considering conditional probabilities given $\mathcal{F}_{n}$) that (4) holds. One idea is to choose $\mathcal{F}_{n}$ so that $B_{n}$ has a very well understood distribution, such as a binomial one.

In some contexts, this approach works directly. Here (as far as we can see) it does not. We can decompose $Q_{n}$ as above with $B_{n}$ binomial (see Lemma 12), but $B_{n}$ will have variance $\Theta(n)$, whereas $\operatorname{Var}Q_{n}=\Theta(n^{2})$. This would, roughly speaking, allow us to establish that ${\mathbb{P}}(Q_{n}=x_{n})$ and ${\mathbb{P}}(Q_{n}=x_{n}^{\prime})$ are similar for $x_{n}-x_{n}^{\prime}=o(\sqrt{n})$, but we need this relation for all $x_{n}-x_{n}^{\prime}=o(n)$.111Actually, since [6] already contains a ‘semi-LLT’, it would suffice to consider $x_{n}-x_{n}^{\prime}=O(n^{5/6})$.

The key idea, as in the draft of [2], is not to try to jump straight from the global limit theorem to the local one, but to proceed in stages.222A related idea has recently been used (independently) by Diaconis and Hough [4], in a different context. They work with characteristic functions, rather than directly with probabilities as we do here, establishing smoothness at a range of frequency scales. For certain pairs of values $\ell<m$ with $\ell\geqslant 1$ and $m=o(n)$ we attempt to show that for any two length-$\ell$ subintervals $I_{1}$, $I_{2}$ of an interval $J$ of length $m$ we have

[TABLE]

The distributional limit theorem gives us that for some $m=o(n)$ each interval $J$ of length $m$ has about the right probability, and we then use the relation above to transfer this to shorter and shorter scales, eventually ending with $\ell=1$. In establishing (5), the idea is as before to find a suitable decomposition $Q_{n}=A_{n}+B_{n}$, but we can use a different decomposition for each scale—there is no requirement that these decompositions be ‘compatible’ in any way. For each pair $(\ell,m)$ we need such a decomposition where the distribution of $B_{n}$ has a property analogous to (5).

There are some complications carrying this out. Our random variables $B_{n}$ will have smaller variances than the original random variables $Q_{n}$. This means that the point probabilities ${\mathbb{P}}(B_{n}=x_{n})$, and (as it turns out) their differences ${\mathbb{P}}(B_{n}=x_{n})-{\mathbb{P}}(B_{n}=x_{n}^{\prime})$, are too large compared with the bounds we are aiming for, and the same holds with the points $x_{n}$ and $x_{n}^{\prime}$ replaced by intervals. For this reason we mostly work with ratios, showing under suitable conditions that ${\mathbb{P}}(B_{n}\in I_{1})\sim{\mathbb{P}}(B_{n}\in I_{2})$. But this is not always true: Even if $I_{1}$ and $I_{2}$ are close, if both are far into a tail of $B_{n}$ the ratio of the probabilities may be far from $1$. To deal with this we use another trick: If for some interval $I_{1}$ there is a significant probability $p$ that $A_{n}+B_{n}\in I$ with the translated interval $I-A_{n}$ being far above the mean of $B_{n}$, say, then there is another interval $J$ (to the left of $I$) such that there is a probability much larger than $p$ that $A_{n}+B_{n}\in J$. Hence what we will actually show, for a series of scales $m$, is that (i) each interval of length $m$ has about the right $Q_{n}$-probability, and (ii) no interval of length $m$ has $Q_{n}$-probability much larger than it should. We will use (ii) at the longer scale $m$ to show that the ‘tail contributions’ described above are small at scale $\ell$. Thus we will be able to transfer the combined statement (i)+(ii) from longer to shorter scales.

In the particular context of QuickSort there is a very nice way to find binomial-like smooth parts: we partially expand the execution tree, looking, roughly speaking, for a way of writing the original instance as the union of $\Theta(s)$ instances of QuickSort each run on $\Theta(r)$ input values, where $s=n/r$. Conditioning on this partial expansion (plus a little further information) the unknown part of the distribution is then ‘binomial-like’: it is a sum of $\Theta(s)$ independent random variables each with ‘scale’ $\Theta(r)$.

The rest of the paper is organized as follows: in Section 2 we state two standard results we shall need later, and then establish the existence of the decompositions described in the previous paragraph. In Section 3 we prove some simple properties of ‘binomial-like’ distributions. Section 4 is the heart of the paper; here we present the core smoothing argument, showing how to transfer ‘smoothness’ from a scale $m$ to a scale $\ell\leqslant m$ under suitable conditions. In Section 5 we complete the proof of Theorem 1; this is a matter of applying the results from Section 4 with suitable parameters, taking as a starting point the ‘semi-local’ limit theorem established by Fill and Janson [6]. Finally, in Section 6 we outline a different way of applying the same smoothing results, taking a weaker distributional convergence result as the starting point; this may be applicable in other settings.

2 Preliminaries

2.1 Some standard inequalities

We shall use the Azuma–Hoeffding inequality (see [1] and [7]) in the following form (see, for example, Ross [14, Theorem 6.3.3]).

Theorem 2**.**

Let $(Z_{n})_{n\geqslant 1}$ be a martingale with mean $\mu=\operatorname{\mathbb{E}{}}Z_{n}$. Let $Z_{0}=\mu$ and suppose that for nonnegative constants $\alpha_{i},\beta_{i}$, $i\geqslant 1$, we have

[TABLE]

Then, for any $n\geqslant 0$ and $a\geqslant 0$ we have

[TABLE]

and the same bound applies to ${\mathbb{P}}(Z_{n}-\mu\leqslant-a)$. ∎

We shall also need Esseen’s inequality, also known as the Berry–Esseen Theorem; see, for example, Petrov [10, Ch. V, Theorem 3]. We write $\Phi$ for the distribution function of the standard normal random variable.

Theorem 3**.**

Let $Z_{1},\ldots,Z_{t}$ be independent random variables with $\rho=\sum_{i=1}^{t}\operatorname{\mathbb{E}{}}(|Z_{i}|^{3})$ finite, and let $S=\sum_{i=1}^{t}Z_{i}$. Then

[TABLE]

where $\mu$ and $\sigma^{2}$ are the mean and variance of $S$, and $A$ is an absolute constant.∎

2.2 Decomposing the execution tree

In this subsection we shall show that, given a parameter $r$, a single run of QuickSort on a list of length $n$ will, with high probability, involve $\Omega(n/r)$ instances of QuickSort run on disjoint lists of length between $r/2$ and $r$.

Let $2\leqslant r<n$ be integers. We can implement QuickSort on a list of length $n$ in two phases as follows: in the first step of Phase I, pick the random pivot dividing the original list into two sublists of total length $n-1$. In step $t$ of Phase I, if all the current sublists have length at most $r$, do nothing. Otherwise, pick a sublist of length at least $r+1$ arbitrarily, and pick the random pivot in this sublist, dividing its remaining elements into two new sublists. After $n$ steps, we proceed to Phase II, where we simply run QuickSort on all remaining sublists. Let $X_{n,r}$ denote the number of sublists at the end of Phase I that have length between $r/2$ and $r$.

Lemma 4**.**

Let $r\geqslant 20$ be even and $n\geqslant 5r$. Then

[TABLE]

Proof.

We have specified that $r$ be even only for convenience. We have made no attempt to optimize the values of the various constants; these will be irrelevant later.

Running QuickSort in two phases as above, let $T$ be the number of ‘active’ steps in Phase I, i.e., steps in which we divide a sublist into two. Clearly, $T\leqslant n$, the first $T$ steps of Phase I are active, and after $T$ steps we have $T+1$ sublists of total length $n-T$. The idea of the proof is to show that $T$ is very unlikely to be larger than $20n/r$, say, that $\operatorname{\mathbb{E}{}}X_{n,r}$ is of order $n/r$, and that each decision in the first phase of our algorithm alters the conditional expectation of $X_{n,r}$ by at most $1$. The result will then follow from the Azuma–Hoeffding inequality.

Throughout the proof we keep $r\geqslant 20$ fixed. Let

[TABLE]

Observe that if $T\geqslant t_{0}$, then after step $t_{0}$ we have $t_{0}+1$ sublists with total length $<n$. Since at most $10n/r\leqslant t_{0}/2$ of these sublists can have length at least $r/10$, at least $t_{0}/2$ of our sublists have length $<r/10$. Let $N$ be the number of sublists after $t_{0}$ steps that have length less than $r/10$, so we have shown that

[TABLE]

In any step of Phase I, we either do nothing, or randomly divide a list of some length $\ell\geqslant r+1$. In the latter case, the (conditional, given the past) probability of producing a sublist of length $<r/10$ is at most

[TABLE]

since $r\geqslant 20$ and $\ell>r$. It follows that $N$ is stochastically dominated by a binomial distribution with parameters $t_{0}$ and $3/10$, so

[TABLE]

using Theorem 2, or a standard Chernoff bound, for the last step.

Turning to the next part of the argument, as $r$ is fixed throughout, let us write $X_{n}$ for the random variable $X_{n,r}$. We extend the definition of $X_{n}$ to the case $n\leqslant r$ by considering Phase I to end immediately (with one ‘sublist’ of length $n$) in this case. The sequence $(X_{n})$ satisfies the deterministic initial conditions

[TABLE]

and (considering the first step in Phase I as described above) the distributional recurrence relation

[TABLE]

where, on the right, $X_{j}$ and $X^{*}_{j}$ are independent probabilistic copies of $X_{j}$ for each $j=1,\dots,n-1$ and $U_{n}$ is uniformly distributed on $\{1,\dots,n\}$, and is independent of all the $X$ and $X^{*}$ variables. Let

[TABLE]

From (7) we have $\xi_{n}=\frac{2}{n}\sum_{i=0}^{n-1}\xi_{i}$ for $n\geqslant r+1$. It follows that

[TABLE]

and

[TABLE]

(The last equation holds also for $n=r$.) Define $\tilde{\xi}_{n}=\frac{n+1}{r+1}$ for all $n$. Then $\tilde{\xi}_{k-1}+\tilde{\xi}_{n-k}=\tilde{\xi}_{n}$ always. Since

[TABLE]

it follows that if $n\geqslant r+1$ (and so $\xi_{n}=\tilde{\xi}_{n}$), then

[TABLE]

for all $1\leqslant k\leqslant n$.

Let $\mathcal{F}_{t}$ denote the $\sigma$-algebra corresponding to information revealed in the first $t$ steps of Phase I as described above. Define

[TABLE]

so that $(M_{t})_{t=0}^{n}$ is a (Doob) martingale. It follows from (9) that the martingale $(M_{t})$, which has mean $M_{0}=\xi_{n}$ given by (8), satisfies

[TABLE]

for every $t$.

Let $E$ be the event that $X_{n}\leqslant\frac{n}{3r}$. Since

[TABLE]

when $E$ holds we have $X_{n}-\xi_{n}\leqslant-\frac{n}{3r}$. After the first $T$ steps of Phase I, nothing further happens, so $M_{T}=M_{T+1}=\cdots=M_{n}=X_{n}$. Hence, writing $t_{0}=\lceil 20n/r\rceil$ as before, we have

[TABLE]

By (6) and the Azuma–Hoeffding inequality (Theorem 2), it follows that

[TABLE]

where the penultimate inequality holds because $20n/r\leqslant t_{0}\leqslant 21n/r$, since $n/r\geqslant 5$, and the final inequality holds because $e^{-8x/5}+e^{-x/378}\leqslant e^{-x/400}$ for $x\geqslant 5$. ∎

Corollary 5**.**

Let $r\geqslant 20$ be even and $n\geqslant 5r$. Then we may write $Q_{n}=A+B$ where, for some $\sigma$-algebra $\mathcal{F}$, we have that $A$ is $\mathcal{F}$-measurable, and, with probability at least $1-e^{-n/(400r)}$, the conditional distribution of $B$ given $\mathcal{F}$ is the sum of $s=\lceil n/(3r)\rceil$ independent random variables $B_{1},\ldots,B_{s}$ with each $B_{i}$ having the distribution $Q_{r_{i}}$ for some $r_{i}$ with $r/2\leqslant r_{i}\leqslant r$.

Proof.

Run QuickSort in two phases as above, and define $X_{n,r}$ as in Lemma 4. Let $E$ be the event that $X_{n,r}\geqslant s=\lceil n/(3r)\rceil$, so ${\mathbb{P}}(E)\geqslant 1-e^{-n/(400r)}$ by Lemma 4. We now subdivide Phase II into two parts. When $E$ holds, we select $s$ sublists from the end of Phase I with length between $r/2$ and $r$, otherwise we do not select any. In Phase IIa, we run QuickSort on all sublists except the selected ones. In Phase IIb, we run QuickSort on the selected sublists. Take the $\sigma$-algebra $\mathcal{F}$ to be the $\sigma$-algebra corresponding to all the information uncovered in Phases I and IIa, and $A$ to be the total number of comparisons made during Phases I and IIa. Take $B_{i}$, $i=1,\dots,s$, to be (when $E$ occurs) the number of comparisons involved in running QuickSort on the $i$th selected sublist. ∎

2.3 Truncating the summands

The sum of the $B_{i}$ above will roughly serve as our ‘binomial-like’ distribution, but we would like a little more information about it. Knowing that $B_{i}$ has ‘scale’ roughly $r_{i}\approx r$, we shall condition on $|B_{i}-\operatorname{\mathbb{E}{}}B_{i}|$ being at most $2r_{i}$. This will still keep a constant fraction of the variance, while giving us better control on the distribution of the sum of such random variables.

Writing $q_{n}$ for $\operatorname{\mathbb{E}{}}Q_{n}$, for $n\geqslant 1$ let $Q_{n}^{*}=(Q_{n}-q_{n})/n$ denote the centered and normalized form of $Q_{n}$. Since $Q_{n}^{*}$ converges in distribution to $Q$, a distribution with a continuous positive density on ${\mathbb{R}}$, we know that there are constants $n_{0}$ and $c_{1}>0$ such that for all $n\geqslant n_{0}$ we have, say, ${\mathbb{P}}(Q_{n}^{*}\in[-2,-1])\geqslant c_{1}$ and ${\mathbb{P}}(Q_{n}^{*}\in[1,2])\geqslant c_{1}$. Hence, for $n\geqslant n_{0}$,

[TABLE]

and, since ${\mathbb{P}}(Q_{n}^{*}\in I\mid Q_{n}^{*}\in[-2,2])\geqslant c_{1}/1=c_{1}$ for $I=[-2,1]$ and $I=[1,2]$, we have

[TABLE]

Let $W_{n}^{\prime}$ denote the distribution of $Q_{n}^{*}$ conditioned to lie in $[-2,2]$, and let $W_{n}:=W_{n}^{\prime}-\operatorname{\mathbb{E}{}}W_{n}^{\prime}$. Then $|W_{n}|\leqslant 4$ and $\operatorname{Var}W_{n}\geqslant c_{1}$. We will record the consequences for the unrescaled distribution of $Q_{n}$ immediately after the following definition.

Definition 1**.**

Given $r>0$ let $\mathcal{D}_{r}$ denote the set of probability distributions of random variables $X$ with the following properties: $\operatorname{\mathbb{E}{}}X=0$, $|X|\leqslant 4r$, and $\operatorname{Var}X\geqslant c_{1}(r/2)^{2}$.

The calculations above have the following simple consequence: for any $r\geqslant 2n_{0}$ and any $r^{\prime}$ satisfying $r/2\leqslant r^{\prime}\leqslant r$, we have ${\mathbb{P}}(Q_{r^{\prime}}\in[q_{r^{\prime}}-2r^{\prime},q_{r^{\prime}}+2r^{\prime}])\geqslant 2c_{1}$, and the conditional distribution of $Q_{r^{\prime}}$ given this event is of the form $z_{r^{\prime}}+X_{r^{\prime}}$ for some constant $z_{r^{\prime}}$ and some $X_{r^{\prime}}$ with law in $\mathcal{D}_{r}$.

Definition 2**.**

Given $r>0$ and a positive integer $s$, let $\mathcal{B}_{r,s}$ denote the set of $s$-fold convolutions of distributions from $\mathcal{D}_{r}$.

In other words, $X$ has a distribution in $\mathcal{B}_{r,s}$ if we can write $X=X_{1}+\cdots+X_{s}$ where the $X_{i}$ are independent and each has law in $\mathcal{D}_{r}$. The distributions in $\mathcal{B}_{r,s}$ will be the ‘binomial-like’ ones we shall use in the smoothing argument.

Remark**.**

More properly we should write $\mathcal{D}_{r,c_{1}}$ and $\mathcal{B}_{r,s,c_{1}}$ for the classes defined in Definitions 1 and 2. In this paper we need only consider a particular value of $c_{1}$ as at the start of this subsection, but in other contexts one might consider these classes for other values of $c_{1}$. The results below of course extend to this setting.

The next lemma, a simple consequence of Corollary 5, will play a key role in our smoothing arguments.

Lemma 6**.**

There are positive constants $r_{0}$, $c_{2}$ and $c_{3}$ such that the following holds whenever $n$ and $r$ are positive integers with $r$ even and $r_{0}\leqslant r\leqslant c_{2}n$: we may write $Q_{n}=A+B$ where, for some $\sigma$-algebra $\mathcal{F}$, we have that $A$ is $\mathcal{F}$-measurable, and, with probability at least $1-e^{-c_{3}n/r}$, the conditional distribution of $B$ given $\mathcal{F}$ is in the class $\mathcal{B}_{r,s}$, with $s=\lceil c_{2}n/r\rceil$.

Proof.

We start by taking $\mathcal{F}^{\prime}$, $A^{\prime}$, and $B^{\prime}$ to be as in Corollary 5. Let $E^{\prime}\in\mathcal{F}^{\prime}$ be the event that we may write the conditional distribution of $B^{\prime}$ as the sum of independent variables $B_{1}^{\prime},\ldots,B_{t}^{\prime}$, $t=\lceil n/(3r)\rceil$, with $B_{i}^{\prime}$ having (conditionally given $\mathcal{F}^{\prime}$) the distribution of $Q_{r_{i}}$ for some $r/2\leqslant r_{i}\leqslant r$. By Corollary 5 we have ${\mathbb{P}}(E^{\prime})\geqslant 1-e^{-\Omega(n/r)}$. We choose $c_{2}\leqslant c_{1}/6$, and set $s=\lceil c_{2}n/r\rceil$. Note that $c_{2}n/r\geqslant 1$, so $s\leqslant 2c_{2}n/r$. We shall reveal certain extra information as described in a moment. Let $E_{i}$ denote the event that $B_{i}^{\prime}\in[q_{r_{i}}-2r_{i},q_{r_{i}}+2r_{i}]$, and let $E$ denote the event that at least $s$ of the events $E_{i}$ occur. Each event $E_{i}$ has conditional probability at least $2c_{1}$ by (10). Since the $E_{i}$’s are conditionally independent given $\mathcal{F}^{\prime}$, and $c_{1}t\geqslant 2c_{2}n/r\geqslant s$, we see [from ${\mathbb{P}}(E\mid E^{\prime})\geqslant{\mathbb{P}}(\mathrm{Bin}(t,2c_{1})\geqslant c_{1}t)$ and Chernoff’s inequality] that ${\mathbb{P}}(E\mid E^{\prime})\geqslant 1-e^{-\Omega(t)}=1-e^{-\Omega(n/r)}$. Hence ${\mathbb{P}}(E)\geqslant 1-e^{-\Omega(n/r)}$.

The extra information we reveal is as follows: firstly, which $E_{i}$’s occur, and hence whether $E$ occurs. When $E$ does occur, we let $\mathcal{I}$ be the set of the first $s$ indices $i$ such that $E_{i}$ occurs, otherwise we may take $\mathcal{I}=\emptyset$, say. We now reveal the values of all $B_{i}^{\prime}$, $i\notin\mathcal{I}$, and set $B=\sum_{i\in\mathcal{I}}(B_{i}^{\prime}-z_{r_{i}})$. Let $\mathcal{F}\supset\mathcal{F}^{\prime}$ denote the $\sigma$-algebra generated by all information revealed so far. Then $A=Q_{n}-B=A^{\prime}+\sum_{i\in\mathcal{I}}z_{r_{i}}+\sum_{i\notin\mathcal{I}}B_{i}^{\prime}$ is certainly $\mathcal{F}$-measurable. Also, when $E$ occurs, the conditional distribution of $B$ given $\mathcal{F}$ is in $\mathcal{B}_{r,s}$, as required. ∎

3 Properties of binomial-like distributions

In this section we establish some simple properties of distributions in the class $\mathcal{B}_{r,s}$ without aiming for tight bounds. The first property is asymptotic normality, which will give ‘smoothness’ on scales larger than $r$. Here and in what follows all constants are absolute, except in that they may depend on the absolute constant $c_{1}$ in the definition of $\mathcal{D}_{r}$.

Lemma 7**.**

For any random variable $X$ with distribution in $\mathcal{B}_{r,s}$ we have $\operatorname{Var}X=\Theta(r^{2}s)$ and

[TABLE]

where $\Phi$ is the standard normal distribution function, and the implicit constants depend only on the constant $c_{1}$ in Definition 1.

Proof.

Dividing $X$, and each of the $s$ summands $X_{i}$ comprising it, through by $r$, we may assume without loss of generality that $r=1$. Then apply the Berry–Esseen Theorem (Theorem 3 above), noting that each $X_{i}$ is bounded in absolute value by $4$, and so has bounded third moment, and that $\operatorname{Var}X$ is $\Theta(s)$ (under our assumption that $r=1$). ∎

Next we establish a common tail bound for all distributions in the class $\mathcal{B}_{r,s}$.

Lemma 8**.**

There are constants $c>0$ and $C$ such that, for all $X$ with distribution in $\mathcal{B}_{r,s}$, all $t\geqslant 0$, and all $\ell\geqslant r$ we have

[TABLE]

Proof.

The Azuma–Hoeffding inequality gives that, uniformly for $Y$ with distribution in $\mathcal{B}_{r,s}$, we have

[TABLE]

Separately, for any interval $I$ of length $\ell\geqslant r$, we have

[TABLE]

Indeed, writing $Y^{\prime}$ for a Gaussian with the same mean and variance as $Y$, by Lemma 7 we have ${\mathbb{P}}(Y\in I)={\mathbb{P}}(Y^{\prime}\in I)+O(1/\sqrt{s})$. As $Y^{\prime}$ has variance $\Theta(r^{2}s)$ we have ${\mathbb{P}}(Y^{\prime}\in I)=O(\ell/(r\sqrt{s}))$, and the error term is absorbed into the main term by the lower bound on $\ell$. Somewhat surprisingly, the proof can be completed by multiplying these two bounds.

Indeed, in proving the claimed result, adjusting the constants if needed, we may assume that $s$ is even. Then we may write $X=Y+Z$ where $Y$ and $Z$ are independent and have distributions in the class $\mathcal{B}_{r,s/2}$. Let $I=[t,t+\ell]$ with $t\geqslant 0$. Since $X\geqslant t$ implies either $Y\geqslant t/2$ or $Z\geqslant t/2$, we may write

[TABLE]

We bound the first term from above by

[TABLE]

The final quantity is $e^{-\Omega(t^{2}/(r^{2}s))}\times O(\ell/(r\sqrt{s}))$ by (11) and (12). The second term in (13) may be bounded in the same way. ∎

3.1 A tilting lemma

In proving Lemma 7 we applied the Berry–Esseen Theorem to distributions from $\mathcal{B}_{r,s}$; next we shall apply the same result to exponential tilts of these distributions, to prove the following result. In what follows, $c_{1}$ is the constant appearing in Definition 1.

Lemma 9**.**

Let $K>0$ be constant. There exists a constant $C^{\prime}=C^{\prime}(c_{1},K)$ such that the following holds whenever

[TABLE]

Let $X$ be a random variable with distribution in $\mathcal{B}_{r,s}$, and let $I_{1}$ and $I_{2}$ be subintervals, each of length $\ell$, of an interval $J$ of length $m$ with $J\subset[-K\lambda r\sqrt{s},K\lambda r\sqrt{s}]$. Then

[TABLE]

The proof of Lemma 9 will be based on standard exponential tilting arguments similar to those used around Lemma 6.4 in [12]. Let $Y$ be a random variable with bounded support. Then for any $\alpha\in{\mathbb{R}}$ we may define the tilted distribution $\mathcal{L}(Y^{(\alpha)})$ by

[TABLE]

where $\gamma=\gamma(\mathcal{L}(Y),a)=\operatorname{\mathbb{E}{}}e^{\alpha Y}$; here $\mathcal{L}(Y)$ denotes the law, or distribution, of the random variable $Y$.

Before starting the proof of Lemma 9, we establish some elementary properties of tilted versions of distributions with law in the set $\mathcal{D}_{1}$ defined in Definition 1.

Lemma 10**.**

There is a constant $c>0$, depending only on $c_{1}$, such that for any $Y$ with $\mathcal{L}(Y)\in\mathcal{D}_{1}$ and any $\alpha\in[-1,1]$ we have $\operatorname{Var}Y^{(\alpha)}\geqslant c$. Furthermore,

[TABLE]

whenever $|\alpha|\leqslant 1$.

Proof.

The first statement is intuitively clear: we take a distribution whose variance is bounded from below, and ‘distort it’ by a bounded amount, so the variance will still be bounded from below. We spell out a concrete argument, not aiming for the best possible bound.

Let $Y$ have distribution in $\mathcal{D}_{1}$. Then, recalling Definition 1, for any $b>0$ we have

[TABLE]

Take $b=\frac{1}{4}\sqrt{c_{1}}$. Then

[TABLE]

Without loss of generality we may thus assume that

[TABLE]

Since $\operatorname{\mathbb{E}{}}Y=0$ and $Y$ is supported on $[-4,4]$, it follows that

[TABLE]

Since $Y$ is supported on $[-4,4]$, for $|\alpha|\leqslant 1$ we have $\gamma=\operatorname{\mathbb{E}{}}e^{\alpha Y}\leqslant e^{4}$, while $e^{\alpha x}$ is at least $e^{-4}$ for all $x$ in the support of $Y$. Hence,

[TABLE]

Similarly,

[TABLE]

The last two bounds clearly imply a lower bound on $\operatorname{Var}Y$ that depends only on $c_{1}$.

To establish (16), note that

[TABLE]

so by the quotient rule,

[TABLE]

∎

Proof of Lemma 9..

Let $X=X_{1}+\cdots+X_{s}$ have distribution in $\mathcal{B}_{r,s}$. We aim to bound ${\mathbb{P}}(X\in I_{2})/{\mathbb{P}}(X\in I_{1})$, where $I_{1}$ and $I_{2}$ are intervals of length $\ell$ both contained in a common interval $J$ of length $m$. By rescaling (considering $\ell/r$ and $m/r$ in place of $\ell$ and $m$) we may assume without loss of generality that $r=1$.

A simple calculation shows that if we tilt each $X_{i}$ by the same parameter $\alpha$, then the independent sum of $X_{1}^{(\alpha)},\ldots,X_{s}^{(\alpha)}$ has the same distribution as $X^{(\alpha)}$. By Lemma 10 we thus have

[TABLE]

for $\alpha\in[-1,1]$. Let

[TABLE]

for $j=1,2$. Since $r=1$ and $m\geqslant C^{\prime}r=C^{\prime}$ by (14), we have

[TABLE]

using the third assumption in (14) in the last step. Hence, choosing $C^{\prime}$ large enough, we have $|t_{1}|\leqslant cs$. Since $\operatorname{\mathbb{E}{}}X^{(0)}=\operatorname{\mathbb{E}{}}X=0$, it follows from (17) that there is a unique value $a\in[-1,1]$ such that

[TABLE]

Moreover, we have

[TABLE]

From now on we fix this tilting parameter, writing $X_{i}^{\prime}$ for $X_{i}^{(a)}$, and $X^{\prime}$ for the independent sum $X_{1}^{\prime}+\cdots+X_{s}^{\prime}$. As noted above, $X^{\prime}$ has the same distribution as $X^{(a)}$. In other words,

[TABLE]

where $\gamma=\operatorname{\mathbb{E}{}}e^{aX}$ is independent of $x$. Since $I_{1}$ and $I_{2}$ lie in an interval $J$ of length $m$, it follows easily that

[TABLE]

Now by (18) and (14),

[TABLE]

so the $e^{O(am)}$ term is $1+O(am)$. Hence

[TABLE]

It remains to bound the ratio ${\mathbb{P}}(X^{\prime}\in I_{2})/{\mathbb{P}}(X^{\prime}\in I_{1})$.

Like the distribution of $X_{i}$, the distribution of $X_{i}^{\prime}$ is supported on the interval $[-4,4]$. Hence $\operatorname{Var}X_{i}^{\prime}=O(1)$. But by the first part of Lemma 10, $\operatorname{Var}X_{i}^{\prime}\geqslant c$, so $\operatorname{Var}X_{i}^{\prime}=\Theta(1)$. Also, the absolute third moment $\operatorname{\mathbb{E}{}}|X_{i}^{\prime}|^{3}$ is clearly at most $4^{3}=O(1)$. The implicit constants in these estimates depend only on $c_{1}$ and $K$, not on $i$. Hence,

[TABLE]

Let us note for later that since $\lambda\geqslant 1$ and $\lambda m/\sqrt{s}=O(1)$ we have

[TABLE]

and hence

[TABLE]

Let $Z\sim{\rm N}(0,1)$ be a standard normal random variable. By the Berry–Esseen Theorem (Theorem 3), we have

[TABLE]

Now by definition $t_{1}=\mu$, and by assumption $I_{1}$ and $I_{2}$ are contained in an interval of length $m$. Hence, for any $y\in I_{1}\cup I_{2}$ we have

[TABLE]

It follows that for $j=1,2$ we have

[TABLE]

Since $A\rho/\sigma^{3}=O(s^{-1/2})$, we thus have

[TABLE]

The implicit constant here does not depend on $C^{\prime}$. Recalling that $\ell\geqslant C^{\prime}r=C^{\prime}$, choosing $C^{\prime}$ large enough, the $1+O(1/\ell)$ factor here is at least $1/2$, and it follows by dividing the bound for $j=2$ by that for $j=1$ that

[TABLE]

Let $\phi(x)=(2\pi)^{-1/2}e^{-x^{2}/2}$ be the density function of the standard normal variable. Since $\phi(x)=\phi(0)e^{O(x^{2})}$, from (21) we have

[TABLE]

Hence,

[TABLE]

recalling (20). Together, (19), (22) and (23) complete the proof. ∎

4 The core smoothing argument

In this section we prove an ungainly lemma (Lemma 11), which is the heart of the smoothing argument. In this lemma there are many parameters; in the next section we give a simple choice of parameters that allows us to prove Theorem 1. The reason for keeping the greater generality here is that it seems (to us) to give a better picture of why the method works, and may help in applying the method in other contexts.

So far, it has not mattered whether the intervals we consider are open, closed or half-open. However, in the arguments below, at certain points we will need to partition one interval into disjoint intervals of the same type. For this reason, from now on we consider only half-open intervals of the form $(a,b]$.

The following definition is key to our smoothing arguments. Recall that $q_{n}=\operatorname{\mathbb{E}{}}Q_{n}$, and that $(Q_{n}-q_{n})/n\overset{\mathrm{d}}{\to}Q$, where $Q$ has a continuous positive density function $f$ on ${\mathbb{R}}$. Given an integer $n\geqslant 1$ and positive real numbers $m$, $\varepsilon$, and $\Gamma\geqslant 1$, we say that the statement $S(n,m,\varepsilon,\Gamma)$ holds333We could work with a statement $S(n,m,L,\varepsilon,\Gamma)$ where, in condition (i) only, the interval $I$ is restricted to lie within $[q_{n}-L,q_{n}+L]$. In Lemmas 11 and 13, the ‘input’ value of $L$ could be anything (at least $m$), and the ‘output’ value of $L$ would be the same as the input. Nothing would change in the proofs. if

(i) for any half-open interval $I\subset{\mathbb{R}}$ of length $m$ and any $x\in{\mathbb{R}}$ such that $q_{n}+nx\in I$ we have $|{\mathbb{P}}(Q_{n}\in I)-\frac{m}{n}f(x)|\leqslant\varepsilon\frac{m}{n}$, and

(ii) for any half-open interval $I$ of length $m$ we have ${\mathbb{P}}(Q_{n}\in I)\leqslant\Gamma\tfrac{m}{n}$.

Roughly speaking, the fact that $(Q_{n}-q_{n})/n\overset{\mathrm{d}}{\to}Q$ implies that $S(n,m,\varepsilon,\Gamma)$ holds for some $m=m(n)=o(n)$, some $\varepsilon=\varepsilon(n)\to 0$, and some constant $\Gamma$. We seek to show that [property (i) of] $S(n,1,\varepsilon^{\prime},\Gamma^{\prime})$ holds for some slightly larger $\varepsilon^{\prime}=\varepsilon^{\prime}(n)$ and $\Gamma^{\prime}$.

Lemma 11**.**

There exist positive constants $c$, $C^{\prime}$, $C$ and $r_{0}$ such that the following holds. If the statement $S(n,m,\varepsilon,\Gamma)$ holds, $\ell$ divides $m$, and there exist real numbers $r$ and $\lambda\geqslant 1$ such that

[TABLE]

then $S(n,\ell,\varepsilon^{\prime},\Gamma^{\prime})$ holds for any $\varepsilon^{\prime}\geqslant\varepsilon+\Gamma\eta$ and $\Gamma^{\prime}\geqslant\Gamma(1+\eta)$, where

[TABLE]

Of course, we could replace $C$ and $C^{\prime}$ by a single constant $\max\{C,C^{\prime}\}$, but they play very different roles in the proof, so we keep them separate. As we shall see later, the key terms on the right in (25) are the second and third; we can choose $\lambda=\log n$, say, and then the key conditions to keep $\eta$ small are (roughly stated) that

[TABLE]

Proof.

Let $J\subset{\mathbb{R}}$ be any interval of length $m$, and let $I_{1}$ and $I_{2}$ be subintervals of $J$ with length $\ell$ such that ${\mathbb{P}}(Q_{n}\in I_{1})$ is minimal and ${\mathbb{P}}(Q_{n}\in I_{2})$ is maximal. We shall show that

[TABLE]

Assuming this for the moment, let us show that the lemma follows. To establish property (i) of $S(n,\ell,\varepsilon^{\prime},\Gamma^{\prime})$, let $I$ be any interval of length $\ell$, and choose an interval $J$ of length $m$ with $I\subset J$. Let $x$ be such that $q_{n}+nx\in I$ and define $I_{1}$ and $I_{2}$ as above. By definition, ${\mathbb{P}}(Q_{n}\in I_{1})\leqslant{\mathbb{P}}(Q_{n}\in I)\leqslant{\mathbb{P}}(Q_{n}\in I_{2})$. Also, since $J$ can be partitioned into intervals of length $\ell$, by simple averaging we have

[TABLE]

By assumption ${\mathbb{P}}(Q_{n}\in J)$ is within $\varepsilon m/n$ of $f(x)m/n$. From (27) and (28) it follows that ${\mathbb{P}}(Q_{n}\in I_{1})$ and ${\mathbb{P}}(Q_{n}\in I_{2})$, and hence ${\mathbb{P}}(Q_{n}\in I)$, are within $(\varepsilon\ell/n)+(\eta\Gamma\ell/n)\leqslant\varepsilon^{\prime}\ell/n$ of $f(x)\ell/n$, as required.

The argument for property (ii) is very similar. Given any interval $I$ of length $\ell$, find an interval $J$ of length $m$ containing it, and define $I_{1}$ and $I_{2}$ as above. This time, by assumption, ${\mathbb{P}}(Q_{n}\in J)\leqslant\Gamma m/n$, so (by averaging) ${\mathbb{P}}(Q_{n}\in I_{1})\leqslant\Gamma\ell/n$. But then ${\mathbb{P}}(Q_{n}\in I)\leqslant{\mathbb{P}}(Q_{n}\in I_{2})\leqslant(\Gamma\ell/n)+(\eta\Gamma\ell/n)\leqslant\Gamma^{\prime}\ell/n$, as required.

It remains to prove (27). Let $r$ and $\lambda\geqslant 1$ satisfy (24). Increasing $r$ slightly if necessary (and adjusting the constants in the lemma appropriately), we may assume that $r$ is an even integer. Let $s=\lceil c_{2}n/r\rceil$ where $c_{2}$ is as in Lemma 6. Write $Q_{n}=A+B$ where $A$, $B$, and the $\sigma$-algebra $\mathcal{F}$ are as in Lemma 6. The idea is to condition on $\mathcal{F}$ and use the fact that, with very high probability, $B$ has conditional distribution in $\mathcal{B}_{r,s}$ to show that ${\mathbb{P}}(Q_{n}\in I_{1})$ and ${\mathbb{P}}(Q_{n}\in I_{2})$ are similar. Let $\sigma=\sqrt{rn}$. Since $s=\Theta(n/r)$, we have $r\sqrt{s}=\Theta(\sigma)$, so by the first part of Lemma 7, $\operatorname{Var}[B\mid\mathcal{F}]=\Theta(r^{2}s)=\Theta(\sigma^{2})$ (with very high probability). It will be crucial that $m\ll\sigma$, so that $I_{1}$ and $I_{2}$ are not too far apart on the scale over which $B$ varies, but that $\ell\gg r$.

In the following proof, all statements hold provided $c$ is small enough and $C$ is large enough. Let $E$ be the event that the conditional distribution of $B$ given $\mathcal{F}$ is indeed in $\mathcal{B}_{r,s}$, so that by Lemma 6

[TABLE]

Let $E_{1}$ be the event that $E$ occurs and the midpoint of $I_{2}-A$ lies in $[-\lambda\sigma,\lambda\sigma]$. Note that $I_{2}$ is deterministic, and $A$ is $\mathcal{F}$-measurable, so $E_{1}\in\mathcal{F}$. Suppose first that $E_{1}$ occurs. Since $I_{1}$ and $I_{2}$ are both subsets of $J$, an interval of length $m\leqslant\sigma/\lambda\leqslant\lambda\sigma$, we have that $I_{1}-A$ and $I_{2}-A$ are both contained in $\{x:|x|\leqslant 2\lambda\sigma\}$. Note that $2\lambda\sigma=O(\lambda r\sqrt{s})$. Also, $\lambda m/(r\sqrt{s})=O(\lambda m/\sqrt{rn})=O(1)$ by (24). Hence, the conditions of Lemma 9 hold for some constant $K$, provided we take $C^{\prime}\geqslant C^{\prime}(c_{1},K)$. When $E_{1}$ occurs, it thus follows from Lemma 9 that

[TABLE]

where

[TABLE]

Since $E_{1}$ is $\mathcal{F}$-measurable, and $Q_{n}\in I_{i}$ if and only if $B\in I_{i}-A$, taking the expectation of both sides of (30) it follows that

[TABLE]

so

[TABLE]

since ${\mathbb{P}}(Q_{n}\in I_{1})\leqslant\frac{\ell}{m}{\mathbb{P}}(Q_{n}\in J)\leqslant\Gamma\frac{\ell}{n}$.

We now consider the ‘tail case’, where $I_{2}-A$ (and hence $I_{1}-A$) is far from the mean (zero) of $B$. We split this case further according to how far.

Assuming purely for convenience that $\lambda$ is an integer, for each integer $y\geqslant\lambda$ let $E_{2,y}^{+}$ be the event that $E$ occurs and the midpoint of $I_{2}-A$ lies in $(y\sigma,(y+1)\sigma]$. Similarly, let $E_{2,y}^{-}$ be the event that $E$ holds and the midpoint of $I_{2}-A$ lies in $[-(y+1)\sigma,-y\sigma)$. Note that

[TABLE]

that this union is disjoint, and that all the events involved are $\mathcal{F}$-measurable.

Fix some $y\geqslant\lambda$ and suppose that $E_{2,y}^{+}$ holds. (The argument for $E_{2,y}^{-}$ will be essentially identical, of course.) Because $I_{2}$ has length at most $\sigma$, we see that the left-endpoint of $I_{2}-A$ is at least $(y-\frac{1}{2})\sigma\geqslant y\sigma/2$. Hence, by Lemma 8,

[TABLE]

after increasing $C$ and decreasing $c$ if necessary. Let $J_{y}=J-y\sigma$, an interval of length $m$ containing $I_{2}-y\sigma$. Note that (for a given $y$) the interval $J_{y}$ is deterministic. Now, when $E_{2,y}^{+}$ holds, $J_{y}-A$ is an interval of length $m$ contained (recalling $m\leqslant\sigma$) in $[-2\sigma,2\sigma]$. By Lemma 7 it follows that

[TABLE]

The implicit constants here depend on $c$, but not on $C^{\prime}$. Recalling our assumption $m\geqslant C^{\prime}r$, we may thus choose $C^{\prime}$ large enough to ensure that the $O(1/\sqrt{s})$ error term is at most half the main term $\Theta(m/\sigma)=\Theta(m/(r\sqrt{s}))$, so

[TABLE]

Combining (32) and (33), we see that when $E_{2,y}^{+}$ holds, then

[TABLE]

Since $E_{2,y}^{+}$ is $\mathcal{F}$ measurable, it follows that

[TABLE]

using property (ii) of the statement $S(n,m,\varepsilon,\Gamma)$ for the last step. A similar bound holds for $E_{2,y}^{-}$. Since $\sum_{y\geqslant\lambda}e^{-cy^{2}}=O(e^{-c\lambda^{2}})$, summing we conclude that

[TABLE]

Finally, recalling (29),

[TABLE]

From (31), (35), and (36) we conclude that

[TABLE]

for some $\eta$ that satisfies

[TABLE]

recalling for the last step that $\Gamma\geqslant 1$ by assumption and that $\sigma=\sqrt{rn}$ by definition. This completes the proof of (27) and thus of the lemma. ∎

Lemma 11 will do most of the work for us, but there is a snag. In applying it, we need to assume $r\ll\ell$. Since we must have $r\geqslant 1$ this means we cannot hope to get down to $\ell=1$ with this method. The fundamental problem is using the Berry–Esseen Theorem (as in the proof of Lemma 9) to try to get a good bound on the probability that an integer-valued random variable is in an interval of length $1$—since the assumptions of the theorem do not distinguish between intervals such as $[k,k+1]$ (which contains two integers) and $[k-1/2,k+1/2]$ (which contains one), we can’t hope for a good approximation in this way. The solution in this case was outlined near the start of the paper: for this part of the argument we work not with a binomial-like distribution, but with a binomial distribution. Then we can calculate the relevant probabilities directly, avoiding the approximation in the Berry–Esseen Theorem. This is captured in Lemma 13 below, whose proof is a variant of the proof of Lemma 11. Before coming to this lemma, we give the decomposition result that we shall need.

Lemma 12**.**

There are constants $c>0$ and $n_{0}$ such that for any $n\geqslant n_{0}$ we may write $Q_{n}=A+B$ where, for some $\sigma$-algebra $\mathcal{F}$, we have that $A$ is $\mathcal{F}$-measurable, and with probability at least $1-e^{-cn}$, the conditional distribution of $B$ given $\mathcal{F}$ is the binomial distribution $\mathrm{Bin}(\lceil cn\rceil,2/3)$.

Of course, this lemma can be rephrased to say that there are independent random variables $A$ and $B$, with $B\sim\mathrm{Bin}(\lceil cn\rceil,2/3)$, such that with probability $1-e^{-cn}$ we have $Q_{n}=A+B$. We keep the wording above to strengthen the analogy to Lemmas 4 and 6.

Proof.

For $n=3$, QuickSort either needs two comparisons (if the initial pivot happens to be the middle element) or, with probability $2/3$, three comparisons. A simple variant of the proof of Lemma 4 shows that if $c>0$ is small enough, then with probability at least $1-e^{-cn}$ we may partially expand the execution tree of QuickSort run on a list of $n$ elements so as to leave $\lceil cn\rceil$ instances of QuickSort of size $3$. We take $B$ to be the number of comparisons in these instances minus $2\lceil cn\rceil$. ∎

Lemma 13**.**

There exist constants $n_{0}$, $c$ and $C$ such that the following holds. Let $n\geqslant n_{0}$ be an integer and let $m\geqslant 1$ and $\lambda\geqslant 1$ be real numbers such that

[TABLE]

If $S(n,m,\varepsilon,\Gamma)$ holds, then so does $S(n,1,\varepsilon^{\prime},\Gamma^{\prime})$ for any $\varepsilon^{\prime}\geqslant\varepsilon+\Gamma\eta$ and $\Gamma^{\prime}\geqslant\Gamma(1+\eta)$, where

[TABLE]

Proof.

We shall show that whenever $x_{1}$ and $x_{2}$ are integers with $|x_{2}-x_{1}|\leqslant m$, then

[TABLE]

The result then follows roughly as in the proof of Lemma 11; since there is a small twist to deal with non-integer $m$, we briefly outline the argument.

First, to establish property (ii) of $S(n,1,\varepsilon^{\prime},\Gamma^{\prime})$, let $x$ be any integer, and pick an interval $J\subset{\mathbb{R}}$ of length $m$ containing $x$, with $\lceil m\rceil$ integers in $J$. By averaging, there is some $x_{1}\in J$ such that

[TABLE]

using the assumption $S(n,m,\varepsilon,\Gamma)$ in the last step. Applying (39) with $x_{2}=x$ gives ${\mathbb{P}}(Q_{n}=x)\leqslant\Gamma^{\prime}/n$, as required. For property (i), using the same $J$ and the bound ${\mathbb{P}}(Q_{n}\in J)\leqslant(f(y)+\varepsilon)m/n$ where $y=(x-q_{n})/n$ gives ${\mathbb{P}}(Q_{n}=x)\leqslant(f(y)+\varepsilon^{\prime})/n$. For the lower bound, consider an interval $J^{\prime}$ of length $m$ containing $x$, with $\lfloor m\rfloor$ integers in $J$. Then find $x_{2}\in J$ with ${\mathbb{P}}(Q_{n}=x_{2})\geqslant{\mathbb{P}}(Q_{n}^{\prime}\in J)/m$ and apply (39) with $x_{1}=x$.

It remains to prove (39). We follow the proof of Lemma 11, but replacing the distribution of class $\mathcal{B}_{r,s}$ by a binomial distribution $\mathrm{Bin}(s,p)$ where $s=\Theta(n)$ and $p=2/3$. (Any $p$ bounded away from [math] and $1$ would work.) The existence of the relevant decomposition is given by Lemma 12; let $s=\lceil cn\rceil$ where $c$ is as in that lemma, and let $E$ be the event that the conditional distribution of $B$ is indeed $\mathrm{Bin}(s,2/3)$, so ${\mathbb{P}}(E)\geqslant 1-e^{-cn}$.

Suppose that $E$ occurs. Then for $0\leqslant k\leqslant s-1$ we have

[TABLE]

If $s/100\leqslant k\leqslant 99s/100$, say, then it follows that

[TABLE]

Recalling that $s=\Theta(n)$, when $k$ is within $2\lambda\sqrt{n}\leqslant cn/10\leqslant s/10$ of $ps$ this gives ${\mathbb{P}}(B=k+1\mid\mathcal{F})/{\mathbb{P}}(B=k\mid\mathcal{F})=1+O(\lambda/\sqrt{n})$ (uniformly in such $k$). It follows that if $k_{1}$ and $k_{2}$ satisfy $|k_{i}-ps|\leqslant 2\lambda\sqrt{n}$ and $|k_{2}-k_{1}|\leqslant m$, then

[TABLE]

Let $E_{1}$ be the event that $E$ occurs and $x_{2}-A$ is within $\lambda\sqrt{n}$ of the mean $ps$ of $B$. Since $m\leqslant\sqrt{n}/\lambda\leqslant\lambda\sqrt{n}$, this implies that $|(x_{1}-A)-ps|\leqslant 2\lambda\sqrt{n}$, say. Using (41) we see that when $E_{1}$ holds, then

[TABLE]

and it follows [arguing as for (31)] that

[TABLE]

As in the proof of Lemma 11, for integer $y\geqslant\lambda$ let $E_{2,y}^{+}$ be the event that $E$ occurs and $x_{2}-A$ lies in $(ps+y\sigma,ps+(y+1)\sigma]$, where now $\sigma=\sqrt{n}$. By an elementary calculation [using (40) as a starting point, for example], letting $k=x_{2}-A$ and recalling that $s=\Theta(n)$, there exist positive constants $c^{\prime}$ and $c$ such that whenever $E_{2,y}^{+}$ holds then

[TABLE]

As before (compare the definition of $J_{y}$ in the proof of Lemma 11), let $J_{y}$ be an interval of length $m$ containing $x_{2}-y\sigma$. Then, when $E_{2,y}^{+}$ holds, $J_{y}-A$ is an interval of length $m$ contained in $[ps-2\sigma,ps+2\sigma]$, say, and it follows using elementary properties of the binomial distribution that

[TABLE]

It follows that when $E_{2,y}^{+}$ holds, then

[TABLE]

the analogue of (34). The rest of the proof follows exactly that of Lemma 11; we omit the details, noting only that the error terms arise as follows: $e^{-c\lambda^{2}}$ from (43) [just as from (34)], $\lambda m/\sqrt{n}$ from (42), and $ne^{-cn}$ from the probability that $E$ fails (recalling that our error terms are written relative to $\Gamma/n$). ∎

Note that in Lemma 13, there is no error term corresponding to the $r/\ell$ term in Lemma 11, which can be traced back to the approximation error from applying the Berry–Esseen Theorem in Lemma 9.

5 Completing the proof

In this section we prove Theorem 1 using Lemmas 11 and 13, together with the following result of Fill and Janson [5, 6].

Theorem 14**.**

Let $F_{n}$ denote the distribution function of $(Q_{n}-q_{n})/n$ and $f$ the continuous density function of the limiting distribution $Q$. There is a constant $C$ such that for any $x$ and any $n\geqslant 1$ we have

[TABLE]

where $\delta_{n}=2Cn^{-1/6}$. Furthermore, $f$ is differentiable on ${\mathbb{R}}$, and we have

[TABLE]

for all $x\in{\mathbb{R}}$.

The first (main) statement is part of [6, Theorem 6.1]; the second statement is from [5, Theorem 3.3]. Of course, the particular values of the constants will not be relevant here.

Rephrased, (44) says that for any (half-open, as usual) interval $I$ of length $m=\delta_{n}n=2Cn^{5/6}$, we have

[TABLE]

where $x_{I}$ is such that $q_{n}+nx_{I}$ is the midpoint of $I$. This is almost, but not quite, condition (i) of the statement $S(n,m,\varepsilon,\Gamma)$ defined before Lemma 11.

Corollary 15**.**

Let $C$ and $\widetilde{C}$ be as in Theorem 14, and set $C^{\prime}=C+\widetilde{C}C$. If $n$ is large enough then $S(n,m,\varepsilon,\Gamma)$ holds with $m=2Cn^{5/6}$, $\varepsilon=C^{\prime}n^{-1/6}$, and $\Gamma=17$.

Proof.

Let $I$ be any (half-open, as always) interval of length $m$. To establish property (i) of $S(n,m,\varepsilon,\Gamma)$ we must show that

[TABLE]

for any $x$ such that $q_{n}+nx\in I$. Let $x_{I}$ be such that $q_{n}+nx_{I}$ is the midpoint of $I$. Then

[TABLE]

By the Mean Value Theorem and the second bound in (45) we have

[TABLE]

This, (46) and the triangle inequality imply (47).

To establish statement (ii) of $S(n,m,\varepsilon,\Gamma)$ simply note that by (47) and (45) we have

[TABLE]

if $n$ is large enough. ∎

We are now ready to prove Theorem 1.

Proof of Theorem 1.

We will show that the theorem holds with the error term $O(n^{-1-\varepsilon})$ replaced by $O(n^{-1-(1/18)}\log n)$. To do this, it suffices to establish that $S(n,1,\varepsilon^{\prime},\Gamma^{\prime})$ holds for some $\varepsilon^{\prime}=O(n^{-1/18}\log n)$; condition (i) of this statement is exactly (1). We establish this by using

[TABLE]

rounds of smoothing, as we now explain in some detail.

In round $k$, for $1\leqslant k\leqslant K-1$, we will apply Lemma 11 with parameters

[TABLE]

to be specified in a moment. Let the constants $C$ and $C^{\prime}$ be as in Corollary 15. We set

[TABLE]

[TABLE]

and

[TABLE]

Furthermore, we set

[TABLE]

and

[TABLE]

for a constant $\widehat{C}$ to be chosen in a moment. Then we inductively define

[TABLE]

and

[TABLE]

The divisibility condition $\ell_{k}|m_{k}$ and the conditions (24) of Lemma 11 are easily seen to hold for $n$ large enough. Moreover, we have

[TABLE]

(Indeed, given $m_{k}$ and $\ell_{k}$, we have chosen $r_{k}$ to balance these terms, ignoring the slowly varying factor $\lambda=\log n$.) Since the outer two terms in (25) are superpolynomially small, we see that if $\widehat{C}$ is chosen suitably large, then in each application of Lemma 11 we have $\eta\leqslant\eta_{k}$. Since the statement $S(n,m_{1},\varepsilon_{1},\Gamma_{1})$ holds by Corollary 15, applying Lemma 11 $K-1$ times we conclude that $S(n,m_{K},\varepsilon_{K},\Gamma_{K})$ holds.

Note that $m_{K}=\Theta(n^{5/6}2^{-K})=\Theta(n^{1/3})$. In the final round we apply Lemma 13 with

[TABLE]

The conditions (37) hold with room to spare (for $n$ large enough). The quantity $\eta$ appearing in (38) is $O(n^{-1/6}\log n)=o(n^{-1/18})$, so we conclude that $S(n,1,\varepsilon^{\prime},\Gamma^{\prime})$ holds, with $\Gamma^{\prime}=O(\Gamma_{K})$ and $\varepsilon^{\prime}=\varepsilon_{K}+o(n^{-1/18}\Gamma_{K})$.

It remains to estimate $\Gamma_{K}$ and $\varepsilon_{K}$. First,

[TABLE]

so (for $n$ large enough), $\Gamma_{k}\leqslant 18$ for all $k$. Then

[TABLE]

It follows that $\varepsilon^{\prime}=O(n^{-1/18}\log n)$, completing the proof. ∎

6 A softer version

We describe here an argument for a weaker version of Theorem 1. The advantage of this argument is that it requires less as input: only a distributional limit theorem, not one with the explicit rate of convergence in Theorem 14. This may be useful in other contexts.

Theorem 16**.**

Uniformly in integers $x$ we have

[TABLE]

as $n\to\infty$, where $f$ is the continuous probability density function of $Q$.

Proof.

We take as our starting point that $(Q_{n}-q_{n})/n\overset{\mathrm{d}}{\to}Q$, making no assumption about the rate of convergence. We do assume certain properties, immediate from [5, Theorem 3.3], of the density function $f$ of $Q$, namely that $f$ is bounded (by $M$, say) and uniformly continuous on ${\mathbb{R}}$. The only other properties of $Q_{n}$ we use are the decompositions provided by Lemmas 6 and 12. This is all we need to prove Lemmas 11 and 13 exactly as above. The difference to the argument in Section 5 is how we apply these lemmas.

Let $F_{n}$ be the distribution function of the normalized distribution $Q_{n}^{*}=(Q_{n}-q_{n})/n$, and let $F$ be that of $Q$. Since $Q_{n}^{*}\overset{\mathrm{d}}{\to}Q$ and $F$ is continuous, we have $F_{n}\to F$ in sup-norm by Polyà’s theorem (for example, [3, Exercise 4.3.4]). In other words, there is some $\delta(n)\to 0$ such that for all $x$ and $n$ we have

[TABLE]

Let

[TABLE]

Since $\delta(n)^{1/2}\to 0$ and $f$ is uniformly continuous, $\gamma(n)\to 0$ as $n\to\infty$. For any interval $I$ of length $|I|=\delta(n)^{1/2}$, by (48) we have that ${\mathbb{P}}(Q_{n}^{*}\in I)$ is within $2\delta(n)$ of $\int_{I}f(x)\mathrm{d}x$ which, by (49), is within $\gamma(n)|I|$ of $f(x)|I|$ for any $x\in I$. Thus, ${\mathbb{P}}(Q_{n}^{*}\in I)$ is within $[2\delta(n)^{1/2}+\gamma(n)]|I|$ of $f(x)|I|$. Replacing $Q_{n}^{*}$ by $Q_{n}$ and $I$ by $q_{n}+nI$, this says exactly that property (i) of $S(n,m_{0},\varepsilon_{0},\Gamma_{0})$ holds, where $m_{0}=m_{0}(n)=n\delta(n)^{1/2}$ and $\varepsilon_{0}=\varepsilon_{0}(n)=2\delta(n)^{1/2}+\gamma(n)$. Taking $\Gamma_{0}=M+1$, which is at least $M+\varepsilon_{0}(n)$ for $n$ large enough, we also have property (ii).

To summarize, the distributional limit theorem (plus assumptions on $f$) gives us that there exist $m_{0}=o(n)$, $\varepsilon_{0}=o(1)$ and $\Gamma_{0}=O(1)$ such that $S(n,m_{0},\varepsilon_{0},\Gamma_{0})$ holds. We now aim to apply Lemma 11 as many times as necessary. The key point is that if $S(n,m,\varepsilon,\Gamma)$ holds where $m=n/\omega$, with $\omega\to\infty$, then in one step we can roughly square $\omega$. More precisely, set $\ell=n/\omega^{1.5}$, say. Then to satisfy (26) we can take say $r=n/\omega^{1.8}$, so

[TABLE]

Choosing $\lambda=\log\omega$, say, the $e^{-c\lambda^{2}}$ term in (25) is superpolynomially small in $\omega$. The term $(n/\ell)e^{-cn/r}$ is $\omega^{1.5}e^{-c\omega^{1.8}}$, which tends to zero extremely quickly as $\omega$ grows. The conclusion is that the conditions of Lemma 11 will hold, with $\eta=O(\omega^{-0.1}\log\omega)=O(\omega^{-0.05})$, say. Applying the lemma repeatedly, in the $i$th application we have $\omega=\omega_{i}=\omega_{0}^{1.5^{i}}$; we stop when $\ell$ is no more than $n^{0.4}$, say (and hence, provided $\omega_{0}\leqslant n^{0.9}$, which we may presume without loss of generality, is at least $n^{0.1}$). Since $\sum_{i}\omega_{i}^{-0.05}=o(1)$ (recalling that $\omega_{0}\to\infty$), the sum of the error terms $\eta$ is $o(1)$, and we find that some $S(n,m,\varepsilon,\Gamma)$ holds with $n^{0.1}\leqslant m\leqslant n^{0.4}$, $\varepsilon=o(1)$ and $\Gamma=O(1)$. A single application of Lemma 13, say with $\lambda=\log n$, yields $S(n,1,\varepsilon^{\prime},\Gamma^{\prime})$ where also $\varepsilon^{\prime}=o(1)$, completing the proof. ∎