Distance-Sensitive hashing

TL;DR

This paper introduces distance-sensitive hashing (DSH), a generalization of LSH, enabling more flexible collision probability functions for high-dimensional data, with applications including privacy-preserving distance estimation.

Contribution

The paper formalizes the DSH framework, allowing for asymmetric hash functions and a wider range of collision probability functions, addressing limitations of traditional LSH.

Findings

Exponential decay of collision probability functions in many spaces

Asymmetry enables increasing or unimodal CPFs

Application to privacy-preserving distance estimation

Abstract

Locality-sensitive hashing (LSH) is an important tool for managing high-dimensional noisy or uncertain data, for example in connection with data cleaning (similarity join) and noise-robust search (similarity search). However, for a number of problems the LSH framework is not known to yield good solutions, and instead ad hoc solutions have been designed for particular similarity and distance measures. For example, this is true for output-sensitive similarity search/join, and for indexes supporting annulus queries that aim to report a point close to a certain given distance from the query point. In this paper we initiate the study of distance-sensitive hashing (DSH), a generalization of LSH that seeks a family of hash functions such that the probability of two points having the same hash value is a given function of the distance between them. More precisely, given a distance space $(X, \text{dist})$ and a "collision probability function" (CPF) $f\colon \mathbb{R}\rightarrow [0,1]$ we seek a distribution over pairs of functions $(h,g)$ such that for every pair of points $x, y \in X$ the collision probability is $\Pr[h(x)=g(y)] = f(\text{dist}(x,y))$. Locality-sensitive hashing is the study of how fast a CPF can decrease as the distance grows. For many spaces, $f$ can be made exponentially decreasing even if we restrict attention to the symmetric case where $g=h$. We show that the asymmetry achieved by having a pair of functions makes it possible to achieve CPFs that are, for example, increasing or unimodal, and show how this leads to principled solutions to problems not addressed by the LSH framework. This includes a novel application to privacy-preserving distance estimation. We believe that the DSH framework will find further applications in high-dimensional data management.

Figures

Equations

$$\ln (1/f(\alpha ))=\frac{1+\alpha }{1-\alpha }\frac{t^2}{2}+\Theta (\log t).$$

$$h:\mathbf{x}\mapsto \lfloor \frac{⟨\mathbf{a},\mathbf{x}⟩+b}{w}\rfloor ,g:\mathbf{x}\mapsto \lfloor \frac{⟨\mathbf{a},\mathbf{x}⟩+b}{w}\rfloor +k,$$

$$\ln \frac{1}{f(\alpha )}=\frac{1-\alpha }{1+\alpha }\ln d+O_{\alpha }(\ln \ln d).$$

$$\ln \frac{1}{f(\alpha )}=\frac{1+\alpha }{1-\alpha }\ln d+O_{\alpha }(\ln \ln d).$$

$$h_{+}(\mathbf{x})$$

$$g_{+}(\mathbf{x})$$

$$g_{-}(\mathbf{x})=g_{+}(-\mathbf{x})=\min (\{i\mid ⟨{\mathbf{z}}_i,\mathbf{x}⟩\le -t\}\cup \{m+2\}).$$

$${\mathbf{y}}_i=\{\begin{array}{ll}{\mathbf{x}}_i& \text{with probability }\frac{1+\alpha }{2},\\ 1-{\mathbf{x}}_i& \text{with probability }\frac{1-\alpha }{2}.\end{array}$$

$$\underset{\begin{array}{c}(\mathbf{x},\mathbf{y})\\ \alpha \text{-corr.}\end{array}}{\mathrm{Pr}}[\mathbf{x}\in A,\mathbf{y}\in B]\ge \exp \left(-\frac{1}{2}\frac{a^2+2\alpha ab+b^2}{1-{\alpha }^2}\right).$$

$$\hat{f}(\alpha )=\underset{\begin{array}{c}(h,g)\sim \mathcal{D}\\ \text{(x,y) α-corr.}\end{array}}{\mathrm{Pr}}[h(\mathbf{x})=g(\mathbf{y})].$$

$$\sum _i(p_i{q}_i{)}^c\ge {\left(\sum _i{p}_i{q}_i\right)}^{2c-1}$$

$$\sum _i(p_i{q}_i{)}^c=\sum _i{p}_i(p_i^{1-1/c}{q}_i)\ge {\left(\sum _i{p}_i^{2-1/c}{q}_i\right)}^c\ge {\left(\sum _i{p}_i{q}_i\right)}^{2c-1}.$$

$$\hat{f}(\alpha )$$

$$\stackrel{(1)}{\ge }{E}_{(h,g)\sim \mathcal{D}}\left[\sum _{i\in R}\exp \left(-\frac{1}{2}\frac{a_{h,i}^2+2\alpha a_{h,i}{b}_{g,i}+b_{g,i}^2}{1-{\alpha }^2}\right)\right]$$

$$\stackrel{(2)}{\ge }{E}_{(h,g)\sim \mathcal{D}}\left[\sum _{i\in R}\exp \left(-\frac{1}{2}\frac{a_{h,i}^2+b_{g,i}^2}{1-\alpha }\right)\right]$$

$$\stackrel{(3)}{\ge }{E}_{(h,g)\sim \mathcal{D}}{\hat{f}}_{h,g}(0{)}^{\frac{1+\alpha }{1-\alpha }}\stackrel{(4)}{\ge }{\left(E_{(h,g)\sim \mathcal{D}}{\hat{f}}_{h,g}(0)\right)}^{\frac{1+\alpha }{1-\alpha }}$$

$$=\hat{f}(0{)}^{\frac{1+\alpha }{1-\alpha }}.$$

$${\rho }_{-}=\frac{\log (1/f_{-})}{\log (1/f_{+})}\ge \frac{1-{\alpha }_{+}}{1+{\alpha }_{+}-2{\alpha }_{-}}-O(\sqrt{\log (1/f_{+})/d}).$$

$$\rho (\mathcal{A})=\frac{\log 1/p}{\log 1/q}\ge \frac{1}{2c-1}-O(\sqrt{(c/r)\log (1/q)}).$$

$$\underset{(\mathbf{x},\mathbf{y})0\text{-corr.}}{\mathrm{Pr}}[\text{dist}(\mathbf{x},\mathbf{y})\le r]\le \exp \left(-\frac{{\epsilon }_p^2}{1-{\epsilon }_p}\frac{r}{2}\right).$$

$$\mathrm{Pr}[X\ge (1+\epsilon )\mu ]\le e^{-{\epsilon }^2\mu /3}.$$

$$(1+{\epsilon }_q)\mu \ge (1+{\epsilon }_q)\frac{2r}{1-{\epsilon }_p}\frac{1-\alpha }{2}\ge r/c.$$

$${\delta }_q\le \exp (-{\epsilon }_q^2\mu /3)\le \exp \left(-\frac{{\epsilon }_q^2}{1+{\epsilon }_q}\frac{r}{3c}\right).$$

$$\hat{q}\le (1-{\delta }_q)q+{\delta }_q.$$

$$\hat{p}$$

$$\hat{q}$$

$$0$$

$${\delta }_p$$

$$\alpha$$

$$\frac{\ln (1/p)}{\ln (1/q)}$$

$$\ge \frac{\ln (1/\hat{p})}{\ln (1+{\delta }_q/q)+\ln (1/\hat{q})}-2{\delta }_p$$

$$\ge \frac{\ln (1/\hat{p})}{\ln (1/\hat{q})}\left(1-\frac{\ln (1+{\delta }_q/q)}{\ln (1/\hat{q})}\right)-2{\delta }_p$$

$$\ge \frac{\ln (1/\hat{p})}{\ln (1/\hat{q})}-\frac{\ln (1+{\delta }_q/q)}{\ln (1/(1+{\delta }_q/q)q)}-2{\delta }_p$$

$$\ge \frac{\ln (1/\hat{p})}{\ln (1/\hat{q})}-2{\delta }_q/q-2{\delta }_p.$$

$$1>\frac{\ln (1/p)}{\ln (1/q)}\ge \frac{\ln (1/\hat{p})}{\ln (1/\hat{q})}.$$

$$\frac{\ln (1/\hat{p})}{\ln (1/\hat{q})}$$

$$\ge \frac{1}{2c-1}-\frac{{\epsilon }^{\prime }}{(2c-1{)}^2}-\frac{{\epsilon }^{\prime }}{2c-1}.$$

Full text

In this paper we initiate the study of distance-sensitive hashing (DSH), a generalization of LSH that seeks a family of hash functions such that the probability of two points having the same hash value is a given function of the distance between them. More precisely, given a distance space $(X,\textnormal{dist})$ and a “collision probability function” (CPF) $f\colon\mathbb{R}\rightarrow[0,1]$ we seek a distribution over pairs of functions $(h,g)$ such that for every pair of points $\mathbf{x},\mathbf{y}\in X$ the collision probability is $\Pr[h(\mathbf{x})=g(\mathbf{y})]=f(\text{dist}(\mathbf{x},\mathbf{y}))$. Locality-sensitive hashing is the study of how fast a CPF can decrease as the distance grows. For many spaces, $f$ can be made exponentially decreasing even if we restrict attention to the symmetric case where $g=h$. We show that the asymmetry achieved by having a pair of functions makes it possible to achieve CPFs that are, for example, increasing or unimodal, and show how this leads to principled solutions to problems not addressed by the LSH framework. This includes a novel application to privacy-preserving distance estimation. We believe that the DSH framework will find further applications in high-dimensional data management.

To put the running time bounds of the proposed constructions into perspective, we show lower bounds for the performance of DSH constructions with increasing and decreasing CPFs under angular distance. Essentially, this shows that our constructions are tight up to lower order terms. In particular, we extend existing LSH lower bounds, showing that they also hold in the asymmetric setting.

locality-sensitive hashing; similarity search; annulus query

††copyright: none††conference: ; ; ††ccs: Theory of computation Randomness, geometry and discrete structures††ccs: Theory of computation Data structures design and analysis††ccs: Information systems Information retrieval

1. Introduction

The growth of data from a variety of sources that need to be managed and analyzed has made it increasingly important to design data management systems with features that make them robust and tolerant towards noisy data. For example: different texts representing the same object (in data reconciliation), slightly different versions of a string (in plagiarism detection), or feature vectors whose similarity reflects the affinity of two objects (in recommender systems). In data management, such tasks are often addressed using the similarity join operator (Silva et al., 2010).

When data sets are high-dimensional, traditional algorithmic approaches often fail. Fortunately, there are general principles for handling high-dimensional data sets. One of the most successful approaches is the locality-sensitive hashing (LSH) framework by Indyk and Motwani (Indyk and Motwani, 1998), further developed in collaboration with Gionis (Gionis et al., 1999) and Har-Peled (Har-Peled et al., 2012). LSH is a powerful framework for approximate nearest neighbor (ANN) search in high dimensions that achieves sublinear query time, and it has found many further applications. However, for a number of problems the LSH framework is not known to yield good solutions, for example output-sensitive similarity search, and indexes supporting annulus queries returning points at distance approximately $r$ from a query point.

Motivating example. A classical application of similarity search is in recommender systems: Suppose you have shown interest in a particular item, for example a news article $\mathbf{x}$. The semantic meaning of a piece of text can be represented as a high-dimensional feature vector, for example computed using latent semantic indexing (Deerwester et al., 1990). In order to recommend other news articles we might search the set $P$ of article feature vectors for articles that are “close” to $\mathbf{x}$. But in general it is not clear that it is desirable to recommend the “closest” articles. Indeed, it might be desirable to recommend articles that are on the same topic but are not too aligned with $\mathbf{x}$, and may provide a different perspective (Abiteboul et al., 2017).

Discussion. Unfortunately, existing LSH techniques do not allow us to search for points that are “close, but not too close”. In a nutshell: LSH provides a sequence of hash functions $h_{1},h_{2},\dots$ such that if $\mathbf{x}$ and $\mathbf{y}$ are close we have $h_{i}(\mathbf{x})=h_{i}(\mathbf{y})$ for some $i$ with constant probability, while if $\mathbf{x}$ and $\mathbf{y}$ are distant we have $h_{i}(\mathbf{x})=h_{i}(\mathbf{y})$ only with small probability (typically $1/n$, where $n$ upper bounds the number of distant points). As a special case, this paper discusses techniques that allow us to refine the first requirement: If $\mathbf{x}$ and $\mathbf{y}$ are “too close” we would like collisions to occur only with very small probability. At first sight this seems impossible because we will, by definition, have a collision when $\mathbf{x}=\mathbf{y}$. However, this objection is overcome by switching to an asymmetric setting where we work with pairs of functions $(h_{i},g_{i})$ and are concerned with collisions of the form $h_{i}(\mathbf{x})=g_{i}(\mathbf{y})$.

More generally, we initiate the systematic study of the following question: In the asymmetric setting, what is the class of functions $f$ for which it is possible to achieve $\Pr[h(\mathbf{x})=g(\mathbf{y})]=f(\text{dist}(\mathbf{x},\mathbf{y}))$, where the probability is over the choice of $(h,g)$ and $\text{dist}(\mathbf{x},\mathbf{y})$ is the distance between $\mathbf{x}$ and $\mathbf{y}$. We refer to such a function as a collision probability function (CPF). More formally:

Definition 1.1.

A distance-sensitive hashing (DSH) scheme for the space $(X,\text{dist})$ is a distribution $\mathcal{D}$ over pairs of functions $h,g\colon X\to\mathbb{R}$ with collision probability function (CPF) $f\colon\mathbb{R}\rightarrow[0,1]$ if for each pair $\mathbf{x},\mathbf{y}\in X$ and $(h,g)\sim\mathcal{D}$ we have $\Pr[h(\mathbf{x})=g(\mathbf{y})]=f(\text{dist}(\mathbf{x},\mathbf{y}))$.

The theory of locality-sensitive hashing is the study of decreasing CPFs whose collision probability is high for neighboring points and low for far-away points.

1.1. Our contributions

We initiate the systematic study of distance-sensitive hashing (DSH), and in particular we:

• •

Show tight upper and lower bounds on the maximum possible growth rate of CPFs under angular distance. This extends upper and lower bound techniques for locality-sensitive hashing to the asymmetric setting.

• •

Provide several DSH constructions that exploit asymmetry between the functions $g$ and $h$ to achieve non-standard CPFs. For example, for Hamming distance we show how to achieve a CPF that equals any polynomial $\mathcal{P}\colon\{0,\dots,d\}\rightarrow[0,1]$ up to a scaling factor.

• •

Present several motivating applications of DSH: Hyperplane queries, annulus search, spherical range reporting, privacy-preserving distance estimation.

The lower bound for angular distance implies lower bounds for Hamming and Euclidean distance DSH. It also shows that existing (asymmetric) LSH constructions used to search for vectors close to a given hyperplane (Vijayanarasimhan et al., 2014) are near-optimal.

On the upper bound side, our constructions show that asymmetric methods are significantly more expressive than standard, symmetric LSH constructions. Since asymmetric methods are often applicable in cases where symmetric methods are used, it seems relevant to re-assess whether such constructions can be improved, even in settings where a decreasing CPF is desired.

Though our DSH applications do not lead to quantitative running time improvements compared to existing, published ad-hoc solutions, we believe that studying the possibilities and limitations of the DSH framework will help unifying approaches to solving “distance sensitive” algorithmic problems. We now proceed with a more detailed description of our results.

1.1.1. Angular distance

We consider monotonically increasing CPFs for angular distance between vectors on the unit sphere $\mathbb{S}^{d-1}$. It will be convenient to express distances in terms of dot products $\langle{\cdot},{\cdot}\rangle$ rather than by angles or Euclidean distances, with the understanding that there is a 1-1 correspondence between them.

The constructions rely on the idea of “negating the query point”: leveraging state-of-the-art symmetric LSH schemes, we obtain DSH constructions with monotone CPFs by replacing the query point $\mathbf{q}$ by $-\mathbf{q}$ in a symmetric LSH construction. We initially apply this idea in section 2.1 to Cross-Polytope LSH (Andoni et al., 2015) getting an efficient DSH with a monotonically decreasing CPF. We then show that a more flexible result follows with a variant of ideas used in the filter constructions from (Becker et al., 2016; Andoni et al., 2017; Christiani, 2017). The filter based approach contains a parameter $t$ that can be used for fine tuning the scheme. This parameter is exploited in the data structure solving the annulus query problem (see section 6.2). More specifically, the filter based approach gives the following result:

Theorem 1.2.

For every $t>1$ there exists a distance-sensitive family $\mathcal{D}_{-}$ for $(\mathbb{S}^{d-1},\langle{\cdot},{\cdot}\rangle)$ with a monotonically decreasing CPF $f$ such that for every $\alpha\in(-1,1)$ with $|\alpha|<1-1/t$ we have

[TABLE]

The complexity of sampling, storing, and evaluating $(h,g)\in\mathcal{D}_{-}$ is $O(dt^{4}e^{t^{2}/2})$.

Note that in terms of the angle between vectors $f$ is increasing, as desired. A corollary of Theorem 1.2 is that we can efficiently obtain a CPF $f$ such that $\rho_{-}=\ln f(0)/\ln f(\alpha)\leq\tfrac{1-\alpha}{1+\alpha}+o_{t}(1)$. (We use $o_{t}(\cdot)$ to indicate that the function depends only on $t$.) The value $\rho_{-}$ measures the gap between collision probabilities (the larger the gap, the smaller $\rho_{-}$), and is significant in search applications. The construction based on cross-polytope LSH has the same CPF as the construction stated in Theorem 1.2 with $t=\sqrt{2\log d}$.

It turns out that the value $\rho_{-}$ is optimal up to the lower-order term. To show this we consider vectors $\mathbf{x},\mathbf{y}\in\{-1,+1\}^{d}$ that are either [math]-correlated or $\alpha$-correlated (i.e., $\Pr[\mathbf{x}_{i}=\mathbf{y}_{i}]=\tfrac{1+\alpha}{2}$ independently for each $i$). These are unit vectors up to a scaling factor $\sqrt{d}$, and for large $d$ the dot product will be tightly concentrated around the correlation. Since correlation is invariant under linear transformation, we may without loss of generality consider $\mathbf{x},\mathbf{y}\in\{0,1\}^{d}$. In section 3 we show the following lower bound:

Theorem 1.3.

Let $\mathcal{D}$ be a distribution over pairs of functions $h,g\colon\{0,1\}^{d}\to R$, and define $\hat{f}\colon[-1,1]\to[0,1]$ as $\hat{f}(\alpha)=\Pr[h(\mathbf{x})=g(\mathbf{y})]$ where $\mathbf{x},\mathbf{y}$ are randomly $\alpha$-correlated and $(h,g)\sim\mathcal{D}$. Then for every $0\leq\alpha<1$ we have that $\hat{f}(\alpha)\geq\hat{f}(0)^{\frac{1+\alpha}{1-\alpha}}$.

Considering $f(\alpha)=\lim_{d\rightarrow\infty}\hat{f}(\alpha)$, taking logarithms of the final inequality we get that $\rho_{-}=\ln f(0)/\ln f(\alpha)\geq\tfrac{1-\alpha}{1+\alpha}$, as desired. Theorem 1.3 extends standard (symmetric) LSH lower bounds (Motwani et al., 2007; Andoni and Razenshteyn, 2016; O’Donnell et al., 2014) to an asymmetric setting in the following sense: If there exists a too powerful asymmetric LSH family $\mathcal{D}_{+}$, then by negating the query point $\mathbf{q}$ we obtain a family $\mathcal{D}_{-}$ with a monotone CPF that contradicts the statement in Theorem 1.3. Observe that this reasoning cannot be applied to standard LSH bounds since they do not handle the asymmetry we allow in our setting.

The proof of the theorem builds on the Reverse Small-Set Expansion Theorem (O’Donnell, 2014) which, given subsets $A,B\subseteq\{0,1\}^{d}$ and random $\alpha$-correlated vectors $\mathbf{x},\mathbf{y}$, lower bounds the probability of the event $[\mathbf{x}\in A\land\mathbf{y}\in B]$ as a function of $\alpha$, $|A|$ and $|B|$. Through a sequence of inequalities we extend the lower bound for pairs of subsets of space to hold for distributions over pairs of functions that partition space, yielding a surprisingly powerful and simple lower bound.

1.1.2. DSH constructions

In the paper we also presents several specific and general constructions for DSH families, in addition to the constructions on the unit sphere discussed above.

In section 4.1, we apply the negating trick to the well-known bit-sampling approach from (Indyk and Motwani, 1998) to get a simple construction of a DSH family with an increasing CPF for Hamming distance. The idea to obtain a DSH with increasing CPF is to use the bit-sampling approach, but negate the bit that is picked from the query point. We show that for small distances, the “collision probability gap” between points at distance $r$ and $r/c$, which is measured by the value $\rho_{-}=\ln f(r)/\ln f(r/c)$, is worse than if we were to map bit strings onto the unit sphere and use the construction from Theorem 1.2. This can be considered somewhat surprising because the bit-sampling LSH family has an optimal collision probability gap for the collision probability of points at distance $r$ and $cr$, measured by the value $\rho_{+}=\ln f(r)/\ln f(cr)$, for small $r$ (O’Donnell et al., 2014).

The negating trick does not work for Euclidean space, since it is potentially unbounded. Nevertheless, we describe in section 4.2 a DSH construction with a $\rho_{-}$-value asymptotically matching the performance of constructions on the unit sphere. The construction is based on an asymmetric version of the classical LSH family for Euclidean space by Datar et al. (Datar et al., 2004): specifically, for parameters $w\in{\bf\mathbb{R}}$ and $k\in{\bf\mathbb{N}}$, we let

[TABLE]

where $b\in[0,w]$ is uniformly random and $\mathbf{a}\sim\mathcal{N}^{d}(0,1)$ is a $d$-dimensional random Gaussian vector. We show that this method, for a suitable choice of parameters $w$ and $k$, provides a near-optimal collision probability gap. This is surprising, since the classical construction in (Datar et al., 2004) is not optimal as an LSH for Euclidean space (Andoni and Indyk, 2006). An example CPF for the above method is shown in Figure 1. As we can see from this figure, the collision probaiblity decreases rapidly on the left side of the maximum, but decreases more slowly on the right side. In particular, this construction has a CPF that is not monotone but unimodal (i.e., a distribution having a single local maximum).

In section 5, we extend our work by targeting the following natural question: let $\mathcal{P}\left(t\right)$ be a polynomial, does there exist a distance-sensitive hash family with CPF $f(t)=\mathcal{P}\left(t\right)$? We present two general approaches of constructing CPFs on the unit sphere and in Hamming space that cover a wide range of such polynomials. The result for the unit sphere easily follows from asymmetric embeddings previously used by Valiant (Valiant, 2015) to solve the closest pair problem. For Hamming space, we propose an approach based on bit-sampling and polynomial factorization that obtains the desired CPF up to a scaling factor $\Delta\geq 1$ that depends on the roots of the polynomial.

1.1.3. Applications

We briefly describe here some applications of DSH constructions. They are all straightforward given a DSH, but give an idea of the versatility of the framework. We refer to section 6 for formal statements and details.

• •

Hyperplane queries. The problem of searching a set of unit vectors for a point (approximately) closest to a given hyperplane can be solved using a CPF $f$, parameterized by dot product, that peaks at $f(0)$. This approach was previously used in ad-hoc contructions, see (Vijayanarasimhan et al., 2014).

• •

Approximate annulus search. The problem of searching for a point at approximately distance $r$ from a query point can be solved using a CPF that peaks at $r$. Previous solutions to this problem used a different, two-stage filtering approach (Pagh et al., 2017).

• •

Approximate spherical range reporting. Classical LSH data structures are inefficient when many near neighbors need to be found, since each neighbor may have a high collision probability and may be “found” many times. A “step function” CPF that is flat for small distances, and then rapidly decreases implies good output-sensitivity. Again, previous results addressing this problem were ad-hoc (Ahle et al., 2017).

• •

Privacy preserving distance estimation. If two parties want to compute the distance between private vectors, limiting the leakage of other information, secure multi-party computations can be used (see e.g. (Goldreich, 2009)). Using “step function” CPFs we can transform this kind of question into a question about Hamming distance between vectors, for which much more efficient protocols exist (Freedman et al., 2004; De Cristofaro et al., 2010).

1.2. Related work

A recent book by Augsten and Böhlen (Augsten and Böhlen, 2013) surveys algorithms for similarity join, with emphasis on data cleaning. Many of the commonly used algorithms are heuristics with weak theoretical guarantees, especially in high dimensions. Recently, however, a substantial literature has been devoted the theoretical study of similarity search and join in data management, e.g. (Ahle et al., 2016; Hu et al., 2017; Kapralov, 2015; Zhang and Zhang, 2017). All of these papers address particular similarity or distance measures, and their results are not directly comparable to those obtained in this paper. In the following we review selected results from the LSH literature in more detail, referring to (Wang et al., 2014; Alexandr Andoni, 2017) for comprehensive surveys.

For simplicity we consider only LSH constructions that are isometric in the sense that the probability of a hash collision depends only on the distance $\text{dist}(\mathbf{x},\mathbf{y})$. In other words, there exists a CPF $f\colon\mathbb{R}\rightarrow[0,1]$ such that $\Pr[h(\mathbf{x})=h(\mathbf{y})]=f(\text{dist}(\mathbf{x},\mathbf{y}))$. Almost all LSH constructions whose collision probability has been rigorously analyzed are isometric. Notable exceptions are recent data dependent LSH methods such as (Andoni and Razenshteyn, 2015) where the LSH distributions, and thus the collision probabilities, depend on the structure of data.

$\rho$-values. Much attention has been given to optimal so-called $\rho$-values of locality-sensitive hash functions, where we consider non-increasing CPFs. Suppose we are interested in hash collisions when $\text{dist}(\mathbf{x},\mathbf{y})\leq r_{1}$ but want to avoid hash collisions when $\text{dist}(\mathbf{x},\mathbf{y})\geq r_{2}$, for some $r_{2}>r_{1}$. The $\rho$-value of this setting (denoted in this paper with $\rho_{+}$) is the ratio of the logarithms of collision probabilities at distance $r_{1}$ and $r_{2}$, i.e., the real number in $[0,1]$ such that $f(r_{1})=f(r_{2})^{\rho}$. The $\rho$-value determines the performance of LSH-based data structures for the $(r_{1},r_{2})$-approximate near neighbor problem, see (Har-Peled et al., 2012). In many spaces a good upper bound on $\rho$ can be given in terms of the ratio $c=r_{2}/r_{1}$, but in general the smallest possible $\rho$ can depend on $r_{1}$, $r_{2}$, $f(r_{1})$, as well as the number of dimensions $d$. In our applications it will be natural to consider the “dual” $\rho$-value that measures the growth rate that can be achieved when distance increases.

LSHable functions. Charikar in (Charikar, 2002) gave a necessary condition that all CPFs in the symmetric setting must fulfill, namely, $\textnormal{dist}(\mathbf{x},\mathbf{y})=1-\Pr[h(\mathbf{x})=h(\mathbf{y})]$ must be the distance measure of a metric, and more specifically this metric must be isometrically embeddable in $\ell_{1}$. In the asymmetric setting this condition no longer holds since in general $\Pr[h(\mathbf{x})=g(\mathbf{x})]<1$.

Chierichetti et al. considered transformations that can be used to create new CPFs (Chierichetti and Kumar, 2015), and studied the decision problem to verify if there exists an LSH with given pairwise collision probabilities (Chierichetti et al., 2014). The transformations in (Chierichetti and Kumar, 2015, Lemma 7) are considered in a symmetric setting, but the same constructions applied in the asymmetric setting give the following result, proved for completeness in Appendix C.1:

Lemma 1.4.

Let $\{\mathcal{D}_{i}\}_{i=1}^{n}$ be a collection of $n$ distance-sensitive families with CPFs $\{f_{i}\}_{i=1}^{n}$.

1. (a)

There exists a distance-sensitive family $\mathcal{D}_{\text{concat}}$ with CPF $f(x)=\prod_{i=1}^{n}f_{i}(x)$. 2. (b)

Given a probability distribution $\{p_{i}\}_{i=1}^{n}$ over $\{\mathcal{D}_{i}\}$, there exists a distance-sensitive family $\mathcal{D}^{p}$ with CPF $f(x)=\sum_{i=1}^{n}p_{i}f_{i}(x)$.

An example application of Lemma 1.4 is shown in Figure 2.

Interestingly, at least in the symmetric setting, the application of this lemma to a single CPF yields all transformations that are guaranteed to map a CPF to a CPF. Chierichetti et al. (Chierichetti et al., 2017) recently extended the study of CPFs in the symmetric setting to allow approximation, i.e., allowing the collision probability to differ from a target function by a given approximation factor.

Asymmetric LSH. Motivated by applications in machine learning, Vijayanarasimhan et al. (Vijayanarasimhan et al., 2014) presented asymmetric LSH methods for Euclidean space where the CPF is a decreasing function of the dot product $|\langle\mathbf{x},\mathbf{y}\rangle|$. Shrivastava and Li (Shrivastava and Li, 2014) also explored how asymmetry can be used to achieve new CPFs (increasing), in settings where the inner product of vectors is used to measure closeness. Neyshabur and Srebro (Neyshabur and Srebro, 2015) extended this study by showing that the extra power obtained by asymmetry hinges on restrictions on the vector pairs for which we consider collisions: If vectors are not restricted to a bounded region of $\mathbb{R}^{d}$, no nontrivial CPF (as a function of inner product) is possible. On the other hand, if one vector is normalized (e.g. a query vector), the performance of known asymmetric LSH schemes can be matched with a symmetric method. But in the case where vectors are bounded but not normalized, asymmetric LSH is able to obtain CPFs that are impossible for symmetric LSH. Ahle et al. (Ahle et al., 2016) showed further impossibility results for asymmetric LSH applied to inner products, and that symmetric LSH is possible in a bounded domain even without normalization if we just allow collision probability 1 when vectors coincide.

Indyk (Indyk, 2003) showed how asymmetry can be used to enable new types of embeddings. More recently asymmetry has been used in the context of locality-sensitive filters (Andoni et al., 2017; Christiani, 2017) and maps (Christiani and Pagh, 2017). The idea is to map each point $\mathbf{x}$ to a pair of sets $(h(\mathbf{x}),g(\mathbf{x}))$ such that $\Pr[h(\mathbf{x})\cap g(\mathbf{y})\neq\emptyset]$ is constant if $\mathbf{x}$ and $\mathbf{y}$ are close, and very small if $\mathbf{x}$ and $\mathbf{y}$ are far from each other. This yields a nearest neighbor data structure that adds for each vector $\mathbf{x}\in P$ the elements of $h(\mathbf{x})$ to a hash table; a query for a vector $\mathbf{q}$ proceeds by looking up each key in $g(\mathbf{q})$ in the hash table. One can transform such methods into asymmetric LSH methods by using min-wise hashing (Broder, 1997; Broder et al., 1997), see (Christiani, 2017, Theorem 1.4).

Recommender systems. Returning to our motivating example, the topic of getting “interesting” recommendations using nearest neighbor methods is not new. Abbar et al. (Abbar et al., 2013) built a nearest neighbor data structure on a core-set of $P$ to guarantee diverse query results. However, this method effectively discards much of the data set, so may not be suitable in all settings. Indyk (Indyk, 2003) and Pagh et al. (Pagh et al., 2017) proposed data structures for finding the furthest neighbor in Euclidean space, leveraging random projections and using specialized data structures.

Privacy-preserving search. Privacy is an increasing concern in connection with data analytics. Proximity information is potentially sensitive, since it may be used to reveal the source of a data point. Ideally we would like information-theoretical privacy guarantees (Agrawal and Aggarwal, 2001), but the standard technique of adding noise to data does not seem to work well for proximity problems, since adding noise merely shifts distances. Riazi et al. (Riazi et al., 2016) considered answering nearest neighbor queries without leaking the actual distance. They showed that standard LSH approaches can compromise privacy under a “triangulation” attack, however this risk can be reduced by designing an LSH (symmetric) with a CPF that is “flat” in the region of interest of an attacker. However, only rather weak privacy guarantees were provided.

2. Optimal angular DSH

This section describes DSH schemes with monotonically increasing and decreasing CPFs for the unit sphere that match the lower bounds shown in the following section 3. As LSH are monotone schemes too (i.e., CPF decreasing with distance or increasing with similarity), we refer to DSH schemes with the opposite monotonicity as anti-LSH (i.e., CPF increasing with distance or decreasing with similarity).

For notational simplicity, the CPF is expressed as function of the inner product of two points, however it holds for other angular similarity and distance measures: on the unit sphere, inner product is equivalent to the cosine similarity and there is a 1-1 correspondence with Euclidean distance and angular distance. Results on the unit sphere can be extended to $\ell_{s}$-spaces for $0<s\leq 2$ through Rahimi and Recht’s (Rahimi and Recht, 2007) embedding version of Bochner’s Theorem (Rudin, 1990) applied to the characteristic functions of $s$-stable distributions as used in (Christiani, 2017).

We use the notation $\mathcal{D}_{-}$ to denote a family with CPF that is decreasing in the inner product between points, and $\mathcal{D}_{+}$ to denote a family with a CPF that is increasing in the inner product between points. The idea behind the following constructions is to take a standard (symmetric) locality-sensitive hash family $\mathcal{D}_{+}$ for the unit sphere with an increasing CPF, and transform it into a family $\mathcal{D}_{-}$ with a decreasing CPF by introducing asymmetry (i.e, by negating the query value).

2.1. Cross-polytope DSH schemes

Andoni et al. (Andoni et al., 2015) described the following LSH family $\mathcal{CP}$ for the $d$-dimensional unit sphere $\mathbb{S}^{d-1}$: To sample a function $h\sim\mathcal{CP}$, sample a random matrix $A\in\mathbb{R}^{d\times d}$ in which each entry is independently sampled from a standard normal distribution $\mathcal{N}(0,1)$. To compute a hash value of a point $\mathbf{x}$, compute $\hat{\mathbf{x}}=A\mathbf{x}/\left\lVert A\mathbf{x}\right\rVert\in\mathbb{S}^{d-1}$ and map $\mathbf{x}$ to the closest point to $\hat{\mathbf{x}}$ among $\{\pm\mathbf{e_{i}}\}_{1\leq i\leq d}$, where $\mathbf{e_{i}}$ is the $i$-th standard basis vector of $\mathbb{R}^{d}$. Intuitively, a random hash function from $\mathcal{CP}$ applies a random rotation to a point and hashes it to its closest point on the cross-polytope.

To formally define a DSH from $\mathcal{CP}$, we sample $(h_{+},g_{+})\sim\mathcal{CP}_{+}$ by sampling $h\sim\mathcal{CP}$ and set $h_{+}=g_{+}=h$. In (Andoni et al., 2015), Andoni et al. showed the following theorem, here reproduced in terms of inner product similarity.111An inner product of $\alpha\in(-1,1)$ between two vectors on the unit sphere corresponds to Euclidean distance $\tau=\sqrt{2(1-\alpha)}$.

Theorem 2.1 (Theorem 1 in (Andoni et al., 2015)).

Let $f$ be the CPF of hash family $\mathcal{CP}_{+}$. Suppose that $\mathbf{x},\mathbf{y}\in\mathbb{S}^{d-1}$ such that $\langle{\mathbf{x}},{\mathbf{y}}\rangle=\alpha$, where $\alpha\in(-1,1)$. Then,

[TABLE]

To obtain a DSH with a monotonically decreasing CPF (in the similarity), we define the family $\mathcal{CP}_{-}$ consisting of pairs $(h_{-},g_{-})$ as follows: To sample a pair $(h_{-},g_{-})\sim\mathcal{CP}_{-}$, sample a function $h\sim\mathcal{CP}$. For each point $\mathbf{x}\in X$, set $h_{-}(\mathbf{x})=h(\mathbf{x})$ and $g_{-}(\mathbf{x})=h(-\mathbf{x})$. This means that we invert the query point before applying the hash function. Intuitively, we map the point to the point on the cross-polytope that is furthest away after applying the random rotation.

Corollary 2.2.

Let $f$ be the CPF of hash family $\mathcal{CP}_{-}$. Suppose that $\mathbf{x},\mathbf{y}\in\mathbb{S}^{d-1}$ such that $\langle{\mathbf{x}},{\mathbf{y}}\rangle=\alpha$, where $\alpha\in(-1,1)$. Then,

[TABLE]

Proof.

If $\langle{\mathbf{x}},{\mathbf{y}}\rangle=\alpha$, then $\langle{\mathbf{x}},{-\mathbf{y}}\rangle=-\alpha$. Since $h_{-}(\mathbf{x})=g_{-}(\mathbf{y})$ corresponds to $h(\mathbf{x})=h(-\mathbf{y})$, we can apply the result of Theorem 2.1 for similarity threshold $-\alpha$. ∎

2.2. Filter-based DSH schemes

Recent work on similarity search for the unit sphere (Andoni et al., 2014; Andoni and Razenshteyn, 2015; Becker et al., 2016; Christiani, 2017) has used variations of the following technique: Pick a sequence of random spherical caps222A spherical cap is a portion of a sphere cut off by a plane. and hash a point $\mathbf{x}\in\mathbb{S}^{d-1}$ to the index of the first spherical cap in the sequence that contains $\mathbf{x}$. By allowing the spherical cap to have different sizes for queries and updates, it was shown how to obtain space-time tradeoffs for similarity search on the unit sphere that is optimal for random data (Andoni et al., 2017; Christiani, 2017). We obtain a family $\mathcal{D}_{-}$ with a decreasing CPF by taking a standard (symmetric) family with its sequence of spherical caps and introduce asymmetry by negating the query point. Intuitively, this means that for $(h,g)\sim\mathcal{D}_{-}$ we let $h$ use the original sequence of spherical caps while $g$ uses the spherical caps that are diametrically opposite to the ones used by $h$. We therefore get a collision $h(\mathbf{x})=g(\mathbf{y})$ if and only if $\mathbf{x}$ and $\mathbf{y}$ are contained in random diametrically opposite spherical caps. We proceed by describing the family $\mathcal{D}_{+}$ and the modification that gives us the family $\mathcal{D}_{-}$.

The family $\mathcal{D}_{+}$ takes as parameter a real number $t>0$ and an integer $m$ that we will later set as a function of $t$. We sample a pair of functions $(h,g)$ from $\mathcal{D}_{+}$ by sampling $m$ vectors $z_{1},\dots,z_{m}$ where $\mathbf{z}_{i}\sim\mathcal{N}^{d}(0,1)$. The functions $h,g$ map a point $\mathbf{x}\in\mathbb{S}^{d-1}$ to the index $i$ of the first projection $\mathbf{z}_{i}$ where $\langle{\mathbf{z}_{i}},{\mathbf{x}}\rangle\geq t$. If no such projection is found, then we ensure that $h(\mathbf{x})\neq g(\mathbf{x})$ by mapping them to different values. Formally, we set

[TABLE]

We use the idea of negating the query point to obtain a family $\mathcal{D}_{-}$ from $\mathcal{D}_{+}$ by setting:

[TABLE]

The analysis of CPF for $\mathcal{D}_{-}$ and $\mathcal{D}_{+}$ and the proof of Theorem 1.2 are provided in Appendix A.1.

3. Lower bound for monotone DSH

This section provides lower bounds on the CPFs of DSH families in $d$-dimensional Hamming space under the similarity measure $\textnormal{sim}_{H}(\mathbf{x},\mathbf{y})=1-2\left\lVert\mathbf{x}-\mathbf{y}\right\rVert_{1}/d$. These results extend to the unit sphere and Euclidean space through standard embeddings.

Our primary focus is to obtain the lower bound in Theorem 1.3, which holds for a CPF that is decreasing with the similarity. As with our upper bounds for the unit sphere, re-applying the same techniques also yields a lower bound for the case of an increasing CPF in the similarity. The proof combines the (reverse) small-set expansion theorem by O’Donnell (O’Donnell, 2014) with techniques inspired by the LSH lower bound of Motwani et al. (Motwani et al., 2007). The main contribution here is to extend this lower bound for pairs of subsets of Hamming space to our object of interest: distributions over pairs of functions that partition space. We begin by introducing the required tools from (O’Donnell, 2014).

Definition 3.1.

For $-1\leq\alpha\leq 1$ and $\mathbf{x},\mathbf{y}\in\{0,1\}^{n}$ we say that $(\mathbf{x},\mathbf{y})$ is randomly $\alpha$-correlated if $\mathbf{x}$ is uniformly distributed over $\{0,1\}^{n}$ and each component of $\mathbf{y}$ is i.i.d. according to

[TABLE]

The reverse small-set expansion theorem lower bounds the probability that random $\alpha$-correlated points $(\mathbf{x},\mathbf{y})$ end up in a pair of subsets $A,B$ of the Hamming cube, as a function of the size of these subsets. In the following, for $A\subset\{0,1\}^{d}$ we refer to the quantity $|A|/2^{d}$ as the volume of $A$.

Theorem 3.2 (Rev. Small-Set Expansion (O’Donnell, 2014)).

Let $0\leq\alpha\leq 1$. Let $A,B\subseteq\{0,1\}^{d}$ have volumes $\exp(-a^{2}/2)$, $\exp(-b^{2}/2)$, respectively, where $a,b\geq 0$. Then we have that

[TABLE]

We define a probabilistic version of the collision probability function that we will state results for. In Section 3.1 we will apply concentration bounds on the similarity between $\alpha$-correlated pairs of points in order to make statements about the actual CPF. We will use $R$ to denote the range of a family of functions which, without loss of generality, we can assume to be finite.

Definition 3.3 (Probabilistic CPF).

Let $\mathcal{D}$ be a distribution over pairs $h,g\colon\{0,1\}^{d}\to R$. We define the probabilistic CPF $\hat{f}\colon[-1,1]\to[0,1]$ by

[TABLE]

The proof of the lower bound will make use of the following technical inequality that follows from two applications of Jensen’s inequality.

Lemma 3.4.

Let $p,q$ denote discrete probability distributions, then for every $c\geq 1$ we have that

[TABLE]

with reverse inequality for $c\leq 1$.

Proof.

Assume $c\geq 1$. By Jensen’s inequality, using the fact that $x\mapsto x^{c}$ and $x\mapsto x^{2-1/c}$ are convex we have that

[TABLE]

For $c\leq 1$ we have that $x\mapsto x^{c}$ and $x\mapsto x^{2-1/c}$ are concave and the inequality is reversed. ∎

We are now ready to state our main lemma that lower bounds $\hat{f}(\alpha)$ in terms of $\hat{f}(0)$. This immediately implies Theorem 1.3.

Lemma 3.5.

For every $0\leq\alpha<1$ and for every distribution $\mathcal{D}$ over pairs of functions $h,g\colon\{0,1\}^{d}\to R$, we have that $\hat{f}(\alpha)\geq\hat{f}(0)^{\frac{1+\alpha}{1-\alpha}}$.

Proof.

For a function $h\colon\{0,1\}^{d}\to R$ define its inverse image $h^{-1}\colon R\to 2^{\{0,1\}^{d}}$ by $h^{-1}(i)=\{\mathbf{x}\in\{0,1\}^{d}\mid h(\mathbf{x})=i\}$. For a pair of functions $(h,g)\in\mathcal{D}$ and $i\in R$ we define $a_{h,i},b_{g,i}\geq 0$ such that $|h^{-1}(i)|/2^{d}=\exp(-a_{h,i}^{2}/2)$ and $|g^{-1}(i)|/2^{d}=\exp(-b_{g,i}^{2}/2)$. For fixed $(h,g)$ define \hat{f}_{h,g}(\alpha)=\Pr_{\text{(\mathbf{x},\mathbf{y})$$\alpha-corr.}}[h(\mathbf{x})=g(\mathbf{y})]. We obtain a lower bound on $\hat{f}(\alpha)$ as follows:

[TABLE]

Here, (1) is due to Theorem 3.2, (2) follows from the simple fact that $a^{2}+\alpha(a^{2}+b^{2})+b^{2}\geq a^{2}+2\alpha ab+b^{2}$, (3) follows from Lemma 3.4 with $c=1/(1-\alpha)$, and (4) follows from a standard application of Jensen’s Inequality. ∎

3.1. Extending the lower bound

We will now use Lemma 3.5 to together with concentration inequalities to obtain a lower bound on the $\rho_{-}$-value of DSH schemes with a CPF that is decreasing in $\textnormal{sim}_{H}(\mathbf{x},\mathbf{y})$. We introduce the following property:

Definition 3.6.

Let $\mathcal{D}$ be a DSH family for $(X,\textnormal{dist})$ with CPF $f$. We say that $\mathcal{D}$ is $(\alpha_{-},\alpha_{+},f_{-},f_{+})$-decreasingly sensitive (resp., $(\alpha_{-},\alpha_{+},f_{-},f_{+})$-increasingly sensitive) if it satisfies:

• •

For $\alpha\leq\alpha_{-}$, we have $f(\alpha)\geq f_{-}$ (respectively, $f(\alpha)\leq f_{-}$);

• •

For $\alpha\geq\alpha_{+}$, we have $f(\alpha)\leq f_{+}$ (respectively, $f(\alpha)\geq f_{+}$).

We observe that if dist is a distance (resp., similarity) measure, then a decreasingly (resp., increasingly) sensitive DSH scheme corresponds to a standard LSH.

Theorem 3.7.

Let $0<\alpha_{-}<\alpha_{+}<1$ be constants. Then every $(\alpha_{-},\alpha_{+},f_{-},f_{+})$-decreasingly sensitive family $\mathcal{D}$ for $(\{0,1\}^{d},\textnormal{sim}_{H})$ must satisfy

[TABLE]

In the statement of Theorem 3.7 we may replace the properties from Definition 3.6 that hold for every $\alpha\leq\alpha_{-}$ and every $\alpha\geq\alpha_{+}$ with less restrictive versions that hold in an $\varepsilon$-interval around $\alpha_{-},\alpha_{+}$ for some $\varepsilon=o_{d}(1)$. Furthermore, if we rewrite the bound in terms of relative Hamming distances $\delta$ and $\delta/c$ where $\delta,c$ are constants, we obtain a lower bound of $1/(2c-1)-o_{d}(1)$ — an expression that is familiar from known LSH lower bounds (Motwani et al., 2007; Andoni and Razenshteyn, 2016).

We now prove Theorem 3.7 proving an analogous result for a $(r,cr,p,q)$-increasingly sensitive family under Hamming distance. Indeed, a $(\alpha_{-},\alpha_{+},f_{-},f_{+})$-decreasingly sensitive family for $(\{0,1\}^{d},\textnormal{sim}_{H})$ is a $(r,cr,p,q)$-increasingly sensitive family for the space $(\{0,1\}^{d},\textnormal{dist}_{H})$, with $\alpha_{-}=1-r/d$ and $\alpha_{+}=1-cr/d$.

Theorem 3.8.

For every constant $\varepsilon>0$, we have that every $(r,cr,p,q)$-increasingly sensitive family $\mathcal{A}$ for $\{0,1\}^{d}$ under Hamming distance with $r\leq(1-\varepsilon)d/2$ must satisfy

[TABLE]

Proof.

Given $\mathcal{A}$ we define a distribution $\hat{\mathcal{A}}$ over pairs of functions $\hat{h},\hat{g}\colon\{0,1\}^{\hat{d}}\to R$ where $\hat{d}\leq d$ remains to be determined. We sample a pair of functions $(\hat{h},\hat{g})$ from $\hat{\mathcal{A}}$ by sampling $(h,g)$ from $\mathcal{H}$ and setting $\hat{h}(\mathbf{x})=h(\mathbf{x}\circ\mathbf{1})$ and similarly $\hat{g}(\mathbf{x})=g(\mathbf{x}\circ\mathbf{1})$ where $\mathbf{1}$ denotes the $(d-\hat{d})$-dimensional all-ones vector. We will now turn to the process of relating $p$ to $\hat{p}=\hat{f}(0)$ and $q$ to $\hat{q}=\hat{f}(\alpha)$ for $\hat{\mathcal{A}}$.

Let $0<\varepsilon_{p}<1$ and set $\hat{d}=\lceil 2r/(1-\varepsilon_{p})\rceil$. Then by applying standard Chernoff bounds we get

[TABLE]

For convenience, define $\delta_{p}=\exp\left(-\frac{\varepsilon_{p}^{2}}{1-\varepsilon_{p}}\frac{r}{2}\right)$. Then $\hat{p}\geq(1-\delta_{p})p.$

In order to tie $\hat{q}$ to $q$ we consider the probability of $\alpha$-correlated points having distance greater than $r/c$. The expected Hamming distance of $\alpha$-correlated $(\mathbf{x},\mathbf{y})$ in $\hat{d}$ dimensions is $\hat{d}(1-\alpha)/2$. We would like to set $\alpha$ such that the probability of the distance exceeding $r/c$ is small. Let $X$ denote $\textnormal{dist}(\mathbf{x},\mathbf{y})$, then the standard Chernoff bound states that:

[TABLE]

For a parameter $0<\varepsilon_{q}<1$ we set $\alpha$ such that the following is satisfied:

[TABLE]

This results in a value of $\alpha=1-\frac{1-\varepsilon_{p}}{1+\varepsilon_{q}}\frac{1}{c}$ and we observe that

[TABLE]

It follows that

[TABLE]

Let us summarize what we know so far:

[TABLE]

We assume that $0<q<p<1$ and furthermore, without loss of generality we can assume that $q\leq 1/e$ due to the powering technique (see Lemma 1.4(a)). In our derivations we also assume that $\delta_{p}\leq 1/2$ and $\delta_{q}\leq 1/(2e)$ such that $\delta_{q}/q\leq 1/2$. This will later be implicit in the statement of the result in big-O notation. From our assumptions and standard bounds on the natural logarithm we are able to derive the following:

[TABLE]

In equation (3) we use the statement itself combined with our assumptions on $p$ and $q$ to deduce that

[TABLE]

We proceed by lower bounding $\hat{\rho}$. Temporarily define $1-\varepsilon^{\prime}=\frac{1-\varepsilon_{p}}{1+\varepsilon_{q}}$ and observe that

[TABLE]

We have that

[TABLE]

and combining these bounds results in

[TABLE]

We can now set $\varepsilon_{q}=\varepsilon_{p}=K\cdot\sqrt{(c/r)\ln(1/q)}$ for some universal constant $K$ to obtain Theorem 3.7. ∎

3.2. Lower bound for asymmetric LSH

We can re-apply the techniques behind Lemma 3.5 and Theorem 3.7 to state similar results in the other direction where for $\alpha_{-}<\alpha_{+}$ we are interested in upper bounding $f(\alpha_{+})$ as a function of $f(\alpha_{-})$. This is similar to the well-studied problem of constructing LSH lower bounds and our results match known LSH bounds (Motwani et al., 2007; Andoni and Razenshteyn, 2016), indicating that the asymmetry afforded by $\mathcal{D}$ does not help us when we wish to construct similarity-sensitive families with monotonically increasing CPFs. Implicitly, this result already follows from the space-time tradeoff lower bounds for similarity search shown independently by Andoni et al. (Andoni et al., 2017) and Christiani (Christiani, 2017). As with Lemma 3.5, the following theorem by O’Donnell (O’Donnell, 2014) is the foundation of our lower bounds.

Theorem 3.9 (Gen. Small-Set Expansion).

Let $0\leq\alpha\leq 1$. Let $A,B\subseteq\{0,1\}^{d}$ have volumes $\exp(-a^{2}/2)$, $\exp(-b^{2}/2)$ and assume $0\leq\alpha b\leq a\leq b$. Then,

[TABLE]

Lemma 3.10.

For every $0\leq\alpha<1$ and for every distribution $\mathcal{D}$ over pairs of functions $h,g\colon\{0,1\}^{d}\to R$, we have $\hat{f}(\alpha)\leq\hat{f}(0)^{\frac{1-\alpha}{1+\alpha}}$.

We are now ready to state the corresponding result for similarity-sensitive families.

Theorem 3.11.

Let $0<\alpha_{-}<\alpha_{+}<1$ be constants. Then every $(\alpha_{-},\alpha_{+},f_{-},f_{+})$-sensitive family $\mathcal{D}$ for Hamming space $(\{0,1\}^{d},\textnormal{sim}_{H})$ must satisfy

[TABLE]

4. Hamming and Euclidean Space DSH

4.1. Anti-LSH construction in Hamming space

Bit-sampling (Indyk and Motwani, 1998) is one of the simplest LSH families for Hamming space, yet gives optimal $\rho_{+}$-values in terms of the approximation factor (O’Donnell et al., 2014). Its CPF is $f(t)=1-t$, where $t$ is the relative Hamming distance. By using a function pair $(\mathbf{x}\mapsto x_{i},\mathbf{x}\mapsto 1-x_{i})$ where $i\in\{1,\dots,d\}$ is random, we get a simple asymmetric DSH family for Hamming space whose CPF $f(t)=t$ is monotonically increasing in the relative Hamming distance. We refer to this specific family as anti bit-sampling. For anti bit-sampling, we get that $\rho_{-}=\ln f(r)/\ln f(r/c)=\Omega(1/\ln c)$ as soon as the relative Hamming distance $r\in[0,1]$ is smaller than $1/e$. Perhaps surprisingly, anti bit-sampling is not optimal and a better result, with $\rho_{-}=O(1/c)$, follows by the anti-LSH schemes based on cross-polytope hashing and filters for the unit sphere in the following subsection. Similarly, the DSH construction for Euclidean space in section 4.2 gives a value of $O(1/c)$ for $\rho_{-}$.

4.2. A DSH construction in Euclidean space

A simple and elegant DSH family in Euclidean space is given by a natural extension of the LSH family introduced by Datar et al. (Datar et al., 2004), where we project a point onto a line and split this line up into buckets. Let $k$ and $w$ be two suitable parameters to be chosen below. Consider the family $\mathcal{R}_{k,w}$ of pairs of functions $(h,g)$ defined in equation (2), indexed by a uniform real number $b\in[0,w]$ and a $d$-dimensional random Gaussian vector $\mathbf{a}\sim\mathcal{N}^{d}(0,1)$. We have the following result whose proof is provided in Appendix B:

Theorem 4.1.

Let $r_{-}$ and $r$ be two real values such that $0<r_{-}<r$, and let $c=r/r_{-}$. Then there exists a constant $w=w(c)$ such that for each $k$ the family $\mathcal{R}_{k,w}$ satisfies

[TABLE]

The proof uses that for $\mathbf{a}\sim\mathcal{N}^{d}(0,1)$ the inner product $\langle\mathbf{a},(\mathbf{x}-\mathbf{y})\rangle$ is distributed as $\mathcal{N}(0,\Delta)$ for two points $\mathbf{x}$ and $\mathbf{y}$ at distance $\Delta$. For the hash values of $\mathbf{x}$ and $\mathbf{y}$ to collide, the inner product must roughly lie in the interval $[(k-1)w,(k+1)w]$. Because we are free to choose $k$ and $w$, we can move the interval into the tail of the $\mathcal{N}(0,\Delta)$ distribution, where the target distances $r$ and $r/c$ have quadratic influence in the exponents.

5. General constructions

So far we have focused our attention on constructions with monotone CPFs, which just represent one kind of DSH schemes. It is natural to wonder if more advanced CPFs can be obtained. In this section, we provide some results in this direction by describing two constructions yielding a wide class of CPFs. We remark that these general constructions do not seem to provide any improved constructions in the monotone case.

Angular similarity functions

We say that $\text{sim}\colon[-1,1]\rightarrow[0,1]$ is an LSHable angular similarity function if there exists an hash family $\mathcal{S}$ with collision probability function $\text{sim}(\langle{\mathbf{x}},{\mathbf{y}}\rangle)$ for each $\mathbf{x},\mathbf{y}\in\mathbb{S}^{d-1}$. For example, the function $\text{sim}(t)=1-\arccos(t)/\pi$ is LSHable using the SimHash construction of Charikar (Charikar, 2002).

Valiant (Valiant, 2015) described a pair of mappings $\varphi^{\mathcal{P}}_{1},\varphi^{\mathcal{P}}_{2}\colon{\mathbb{R}}^{d}\rightarrow{\mathbb{R}}^{D}$, where $D=O(d^{k})$, such that $\varphi^{\mathcal{P}}_{1}(\mathbf{x})\cdot\varphi^{\mathcal{P}}_{2}(\mathbf{y})=\mathcal{P}\left(\langle{\mathbf{x}},{\mathbf{y}}\rangle\right)$, for any polynomial $\mathcal{P}\left(t\right)=\sum_{i=0}^{k}a_{i}t^{i}$. By leveraging this construction we get the following result (with proof provided in Appendix C.2).

Theorem 5.1.

Suppose that sim is an LSHable angular similarity function and that the polynomial $\mathcal{P}\left(t\right)=\sum_{i=0}^{k}a_{i}t^{i}$ satisfies $\sum_{i=0}^{k}|a_{i}|$ = 1. Then there exists a distribution over pairs $(h,g)$ of functions such that for all $\mathbf{x},\mathbf{y}\in\mathbb{S}^{d-1}$, $\Pr[h(\mathbf{x})=g(\mathbf{y})]=\textnormal{sim}(\mathcal{P}\left(\langle{\mathbf{x}},{\mathbf{y}}\rangle\right))$.

The computational cost of a naïve implementation of the proposed scheme may be prohibitive when $d^{k}$ is large. However, by using the so-called kernel approximation methods (Pham and Pagh, 2013), we can in near-linear time compute approximations $\hat{\varphi}^{\mathcal{P}}_{1}(\mathbf{x})$ and $\hat{\varphi}^{\mathcal{P}}_{2}(\mathbf{y})$ that satisfy $\hat{\varphi}^{\mathcal{P}}_{1}(\mathbf{x})\cdot\hat{\varphi}^{\mathcal{P}}_{2}(\mathbf{y})=\mathcal{P}\left(\langle{\mathbf{x}},{\mathbf{y}}\rangle\right)\pm\varepsilon$ with high probability for a given approximation error $\varepsilon>0$.

Hamming distance functions

It is natural to wonder which CPFs can be expressed as a function of the relative Hamming distance $d_{h}(\mathbf{x},\mathbf{y})$. A first answer follows by using the anti bit-sampling approach from section 4 together with Lemma 1.4. This gives a scheme for matching any polynomial $\mathcal{P}\left(t\right)=\sum_{i=0}^{k}a_{i}t^{i}$ that satisfies $\sum_{i=0}^{k}a_{i}=1$ and $a_{i}>0$ for each $i$.

In this section, we provide another construction that matches, up to a scaling factor $\Delta$, any polynomial $\mathcal{P}\left(t\right)$ having no roots with a real part in $(0,1)$. The scaling factor depends only on the roots of the polynomial. We claim that such a factor is unavoidable in the general case: indeed, without $\Delta$, it would be possible to match the CPF $1-t^{2}$ for Hamming space, which implies $\rho\leq 1/c^{2}$ in contradiction with the lower bound $1/c$ in (O’Donnell et al., 2014). (Nevertheless, it is an open question to assess how tight $\Delta$ is.) We have the following result that is proven in Appendix C.3:

Theorem 5.2.

Let $\mathcal{P}\left(t\right)=\sum_{i=0}^{k}a_{i}t^{i}$, $Z$ be the multiset of roots of $\mathcal{P}\left(t\right)$, and $\psi\leq k$ be the number of roots with negative real part. Then there exists a DSH family with collision probability $\Pr(h(\mathbf{x})=g(\mathbf{y}))=\mathcal{P}\left(d_{h}(\mathbf{x},\mathbf{y})\right)/\Delta$ with $\Delta=a_{k}2^{\psi}\prod_{z\in Z,|z|>1}|z_{i}|$.

The construction exploits the factorization $\mathcal{P}\left(t\right)=a_{k}\prod_{z\in Z}(t-z)$ and consists of a combination of $|Z|$ variations of bit-sampling and anti bit-sampling. Although the proposed scheme may not reach the $\rho$-value given by the polynomial $\mathcal{P}\left(t\right)$, it can be used for estimating $\mathcal{P}\left(d_{H}(\mathbf{x},\mathbf{y})\right)$ since the scaling factor is constant and only depends on the polynomial.

Finally, we observe that our scheme can be used to approximate any function $f(t)$ that can be represented with a Taylor series: indeed, it is sufficient to truncate the series to the term that gives the desired approximation, and then to apply our construction to the resulting truncated polynomial.

6. Applications

6.1. Hyperplane queries and annulus search

Approximate annulus search is the problem of finding a point in the set $P$ of data points with distance in an interval $[r_{-},r_{+}]$ from a query point. On the unit sphere, hyperplane queries are a special case of annulus queries where we want to find a point with inner product close to [math] to a query point. This type of search has applications in machine learning (see (Jain et al., 2010; Liu et al., 2012)).

The ad hoc solution for this problem (Pagh et al., 2017) in Euclidean space works by first building an LSH data structure that aims to retrieve points at distance at most $r$ while filtering out points at distance at least $r_{+}$. In each repetition and in each bucket of the hash table, one builds a data structure that is set up to filter away points at distance at least $r_{-}$ while preserving points at distance at least $r$. The latter filtering can be thought of as applying an anti-LSH in each bucket. For $r_{-}=r/c$ and $r_{+}=cr$ for a $c\geq 1$, the construction in (Pagh et al., 2017) answers queries in time $\tilde{O}(dn^{\rho+1/c^{2}})$ and the data structure uses space $\tilde{O}(n^{1+\rho+1/c^{2}}+dn)$, where $\rho$ describes the collision probability gap of the LSH that is used, disregarding logarithmic terms by using the $\tilde{O}(.)$ notation.

Having access to a DSH family with a CPF that peaks inside $[r_{-},r_{+}]$ and is significantly smaller at the ends of the interval gives an LSH-like solution to this problem.

Theorem 6.1.

Suppose we have a set $P$ of $n$ points, an interval $[r_{-},r_{+}]$, a distance $r\in[r_{-},r_{+}]$, and assume we are given a DSH scheme with a CPF $f$ that peaks inside $[r_{-},r_{+}]$ and satisfies $f(r^{\prime})\leq 1/n$ for all $r^{\prime}\notin[r_{-},r_{+}]$. Then there exists a data structure that, given a query $\mathbf{q}$ for which there exists $\mathbf{x}\in P$ with $\textnormal{dist}(\mathbf{q},\mathbf{x})=r$, returns $\mathbf{x}^{\prime}\in P$ with $\textnormal{dist}(\mathbf{q},\mathbf{x}^{\prime})\in[r_{-},r_{+}]$ with probability at least $1/2$. The data structure uses space $O(n^{1+\rho^{\ast}}/f(r)+dn)$ and has query time $O(dn^{\rho^{\ast}})$, where $\rho^{\ast}=\log(1/f(r))/\log n$.

Proof.

The data structure is a straightforward adaptation of the construction of a near neighbor data structure using LSH. Associate with each data point $\mathbf{x}$ and query point $\mathbf{y}$ the hash values $h(\mathbf{x})$ and $g(\mathbf{y})$, where $(h,g)$ are independently sampled from the distance-sensitive family. Store all points $\mathbf{x}\in S$ according to $h(\mathbf{x})$ in a hash table. Let $\mathbf{y}$ be the query point and let $\mathbf{x}$ be a point at distance $r$. Compute $g(\mathbf{y})$ and retrieve all the points from $S$ that have the same hash value. If a point within distance $[r_{-},r_{+}]$ is among the points, output one such point. We expect $\max\{f(r_{-})n,f(r_{+})n\}\leq 1$ collisions with points at distance at most $r_{-}$ or at least $r_{+}$. The probability of finding $\mathbf{x}$ is at least $f(r)$. Thus, $L=1/f(r)\leq n^{\rho^{\ast}}$ repetitions suffice to retrieve $\mathbf{x}$ with constant probability $1/e$. If the algorithm retrieves more than $8L$ points, none of which is in the interval $[r_{-},r_{+}]$, the algorithm terminates. By Markov’s inequality, the probability that the algorithm retrieves $8L$ points, none of which is in the interval $[r_{-},r_{+}]$, is at most $1/8$. ∎

We note that the assumption $f(r_{+}),f(r_{-})\leq 1/n$ in the theorem is not critical: the standard technique of powering (see Lemma 1.4(a)) allows us to work with the CPF $f(x)^{k}$ for integer $k$, where $k$ is the smallest integer such that $f(x)^{k}\leq 1/n$.

We observe that this data structure improves the trivial scanning solution when $\rho^{\ast}=\log(1/f(r))/\log n<1$, that is when $f(r)>f(r_{-})$ and $f(r)>f(r_{+})$. This is satisfied by unimodal distance-sensitive hash families, that is when the CPF has a single maximum at $t^{\ast}$ and is decreasing for both $t\leq t^{\ast}$ and $t\geq t^{\ast}$: as soon as $t^{\ast}$ lies in the interval $(r_{-},r_{+})$ we obtain a data structure with sublinear query time.

Obtaining a CPF that peaks inside of $[r_{-},r_{+}]$ can be achieved by combining a standard LSH family $\mathcal{H}$ with a DSH family $\mathcal{A}$ that has an increasing CPF by means of powering each part. For example, when combining a bit-sampling and an anti bit-sampling family, concatenating $k_{1}$ bit-sampling and $k_{2}$ anti bit-sampling results in the CPF $f(t)=(1-t)^{k_{1}}t^{k_{2}}$. Setting $k_{1}=k_{2}(1-t)/t$ results in $f$ peaking at distance $r$. In general, for the value $\rho^{\ast}$ in the statement of the theorem, this approach yields a bound of $\rho^{\ast}\leq\rho_{+}+\rho_{-}$, where $\rho_{+}$ and $\rho_{-}$ are the $\rho$-values of $\mathcal{H}$ and $\mathcal{A}$, respectively. This is essentially the same as the running time guarantee of the ad hoc data structure in (Pagh et al., 2017) in Euclidean space.

In section 6.2, we show how to combine the two monotonic constructions from section 2.2 on the unit sphere. A key ingredient of the construction is that we do not need the powering approach described above but can combine a single LSH with a single anti-LSH function and set the thresholds of each part accordingly. With respect to hyperplane queries under inner product similarity, the resulting construction allows us to search a point set $P$ of unit vectors for a vector approximately orthogonal to a query vector $\mathbf{q}$ in time $dn^{\rho^{\ast}+o(1)}$ for $\rho^{\ast}=\tfrac{1-\alpha^{2}}{1+\alpha^{2}}$, where we guarantee to return a vector $\mathbf{x}$ with $\langle\mathbf{x},\mathbf{q}\rangle\in[-\alpha,\alpha]$ if an orthogonal vector exists. When applied to search for approximately orthogonal vectors, our technique improves $\rho$-values of previous techniques, but the improvement is not surprising in view of recent progress in angular LSH, see e.g. (Andoni et al., 2015). However, Theorem 6.4 in Section 6.2 supports searching a wide range of different annuli and not just the ones centered around vectors with zero correlation. As an additional result, we obtain a definition of an annulus in a space with bounded distances.

6.2. Annulus search on the unit sphere

We will construct a distance sensitive family $\mathcal{D}$ for solving the approximate annulus search problem. Let $\mathcal{D}_{+}$ be parameterized by $t_{+}$ and let $\mathcal{D}_{-}$ be parameterized by $t_{-}$. To sample a pair of functions $(h,g)$ from $\mathcal{D}$ we independently sample a pair $(h_{+},g_{+})$ from $\mathcal{D}_{+}$ and $(h_{-},g_{-})$ from $\mathcal{D}_{-}$ and define $(h,g)$ by $h(\mathbf{x})=(h_{+}(\mathbf{x}),h_{-}(\mathbf{x}))$ and $g(\mathbf{x})=(g_{+}(\mathbf{x}),g_{-}(\mathbf{x}))$.

Let $f(\alpha)$ denote the CPF of $\mathcal{D}$. We would like to be able to parameterize $\mathcal{D}$ such that $f(\alpha)$ is somewhat symmetric around a unique maximum value of $\alpha$. It can be verified from the definition of $\mathcal{D}_{+}$ that $p_{+}(-1)=0$ which implies that $f(-1)=f(1)=0$. If we ignore lower order terms and define $\gamma>0$ by $t_{-}=\gamma t_{+}$, then we can see that

[TABLE]

For simplicity, temporarily define $a(\alpha)=(1-\alpha)/(1+\alpha)>0$. Given a fixed $\gamma$, the equation $a+\gamma^{2}/a$ is minimized (corresponding to approximately maximizing $f(\alpha)$) when setting $a=\gamma$. Let $\alpha_{\max}\in(-1,1)$ and set $\gamma=a_{\max}=(1-\alpha_{\max})/(1+\alpha_{\max})$. To find values $\alpha_{-}<\alpha_{\max}<\alpha_{+}$ where $\ln(1/f(\alpha_{-}))\approx\ln(1/f(\alpha_{+}))$ note that this condition holds for every $s>1$ when we set $a_{-}=sa_{\max}$ and $a_{+}=(1/s)a_{\max}$. We therefore parameterize $\mathcal{D}$ by $\alpha_{\max}\in(-1,1)$ and $t>0$ and set $t_{+}=t$ and $t_{-}=(1-\alpha_{\max})/(1+\alpha_{\max})t_{+}$. By combining our bounds from Lemma A.5 with the above observations we are able to obtain the following theorem which immediately yields a solution to the approximate annulus search problem.

Theorem 6.2.

For every choice of $t>0$ and constant $\alpha_{\max}\in(-1,1)$ the family $\mathcal{D}$ satisfies the following: For every choice of constant $s>1$ consider the interval $[\alpha_{-},\alpha_{+}]$ defined to contain every $\alpha$ such that $\frac{1}{s}\frac{1-\alpha_{\max}}{1+\alpha_{\max}}\leq\frac{1-\alpha}{1+\alpha}\leq s\frac{1-\alpha_{\max}}{1+\alpha_{\max}}$, then

• •

For $\alpha\in[\alpha_{-},\alpha_{+}]$ we have that

[TABLE]

• •

For $\alpha\notin[\alpha_{-},\alpha_{+}]$ we have that

[TABLE]

The complexity of sampling, storing, and evaluating a pair of functions $(h,g)\in\mathcal{D}$ is $O(dt^{4}e^{t^{2}/2})$.

See Figure 3 for a visual representation of the annulus for given parameters $\alpha_{\max}$ and $s$.

We define an approximate annulus search problem for similarity spaces and proceed by applying Theorem 6.2 to provide a solution for the unit sphere, resulting in Theorem 6.4.

Definition 6.3.

Let $\beta_{-}<\alpha_{-}\leq\alpha_{+}<\beta_{+}$ be given real numbers. For a set $P$ of $n$ points in a similarity space $(X,\textnormal{sim})$ a solution to the $((\alpha_{-},\alpha_{+}),(\beta_{-},\beta_{+}))$-annulus search problem is a data structure that supports a query operation that takes as input a point $\mathbf{x}\in X$ and if there exists a point $\mathbf{y}\in P$ such that $\alpha_{-}\leq\textnormal{sim}(\mathbf{x},\mathbf{y})\leq\alpha_{+}$ then it returns a point $\mathbf{y}^{\prime}\in P$ such that $\beta_{-}\leq\textnormal{sim}(\mathbf{x},\mathbf{y}^{\prime})\leq\beta_{+}$.

Theorem 6.4.

For every choice of constants $-1<\beta_{-}<\alpha_{-}<\alpha_{+}<\beta_{+}<1$ such that $\frac{1-\alpha_{-}}{1+\alpha_{-}}\frac{1-\alpha_{+}}{1+\alpha_{+}}=\frac{1-\beta_{-}}{1+\beta_{-}}\frac{1-\beta_{+}}{1+\beta_{+}}$ we can solve the $((\alpha_{+},\alpha_{-}),(\beta_{+},\beta_{-}))$-annulus problem for $(\mathbb{S}^{d-1},\langle{\cdot},{\cdot}\rangle)$ with space usage $dn+n^{1+\rho+o(1)}$ words and query time $dn^{\rho+o(1)}$ where

[TABLE]

and we define $1<c_{\alpha}<c_{\beta}$ by c_{\alpha}=\sqrt{\frac{1-\alpha_{-}}{1+\alpha_{-}}\big{/}\frac{1-\alpha_{+}}{1+\alpha_{+}}}, c_{\beta}=\sqrt{\frac{1-\beta_{-}}{1+\beta_{-}}\big{/}\frac{1-\beta_{+}}{1+\beta_{+}}}, and $c=c_{\beta}/c_{\alpha}$.

6.3. Spherical range reporting

Approximate spherical range reporting (Ahle et al., 2017) aims to report all points in $P$ within distance $r$ from a query point. A common problem with LSH-based solutions for reporting all close points is that the CPF is monotonically decreasing starting with collision probability very close to $1$ for points that are very close to the query point. On the other hand, many repetitions are necessary to find points at the target distance $r$. This means that the algorithm retrieves many duplicates for solving range reporting problems. The state-of-the-art data structure for range reporting queries (Ahle et al., 2017) requires $O\left((1+|S^{\ast}|)(n/|S^{\ast}|)^{\rho}\right)$, where $S^{\ast}$ is the set of points at distance at most $r_{+}$.

CPFs that have a (roughly) fixed value in $[0,r]$ and then decrease rapidly to zero (so-called “step-function CPFs”) yield data structures with good output sensitivity.

Theorem 6.5.

Suppose we have a set $P$ of $n$ points and two distances $r<r_{+}$. Assume we are given a DSH scheme with CPF $f$ where $f(r^{\prime})\leq 1/n$ for all $r^{\prime}\geq r_{+}$, and let $f_{\textnormal{min}}=\inf_{t\in[0,r]}f(t)$, $f_{\textnormal{max}}=\sup_{t\in[0,r]}f(t)$. Then there exists a data structure that, given a query $\mathbf{q}$, returns $S\subseteq\{\mathbf{x}\in P\mid\textnormal{dist}(\mathbf{q},\mathbf{x})\leq r_{+}\}$ such that for each $\mathbf{x}\in P$ with $\textnormal{dist}(\mathbf{q},\mathbf{x})\leq r$, $\Pr[\mathbf{x}\in S]>1/2$. The data structure uses space $O(n^{1+\rho^{\ast}}+dn)$ and the query has expected running time $O(dn^{\rho^{\ast}}+d|S|f_{\textnormal{max}}/f_{\textnormal{min}})$, where $\rho^{\ast}=\log(1/f_{\textnormal{min}})/\log(1/f(r_{+}))$.

Proof.

We assume that we build a standard LSH data structure as in the proof of Theorem 6.1 above. We use $1/f_{\min}$ repetitions such that each point within distance $r$ is found with constant probability. Each repetition will contribute $O\left(1+|S^{\ast}|f_{\max}\right)$ points in expectation. Thus, the total cost will be $O\left((1+|S^{\ast}|f_{\max})/f_{\min}\right)$ from which the statement follows. ∎

In short, Theorem 6.5 provides a better analysis of the performance of a standard LSH data structure that takes into account the gap between $f_{\min}$ and $f_{\max}$. Again, the assumption $f(r_{+})\leq 1/n$ in the theorem is not critical.

In particular, the theorem shows that if we have a constant bound on $f_{\text{max}}/f_{\text{min}}$ the output sensitivity is optimal in the sense that the time to report an additional close point is $O(d)$ which is the time it takes to verify its distance to the query point. Such step-function CPFs are implicit in the linear space extremes of the space-time tradeoff techniques for near neighbor search (Andoni et al., 2017; Christiani, 2017). To see why this is the case consider a randomized (list-of-points (Andoni et al., 2017)) data structure that solves the approximate near neighbor problem using linear space. The data structure stores each point in exactly one bucket. During a query $L$ different buckets are searched. The data structure has the property that near neighbors collide in at least one bucket with constant positive probability. We can now construct a family $\mathcal{D}$ where we sample $(h,g)\sim\mathcal{D}$ by sampling a random (data-independent) data structure. We set $h(\mathbf{x})$ equal to the index of the single bucket that $\mathbf{x}$ would be stored in during an update, and $g(\mathbf{x})$ is set to the index of a random bucket out of the $L$ buckets that would be searched during a query for $\mathbf{x}$. If the data structure guarantees finding an $r$-near neighbor, then the probability of collision is $\Theta(1/L)$ for points $\mathbf{x},\mathbf{y}$ with $\textnormal{dist}(\mathbf{x},\mathbf{y})\leq r$. Even though this theoretical construction gives optimal output sensitivity, it is possible that a better value of $\rho^{*}$ can be obtained by allowing a higher space usage.

6.4. Privacy-Preserving Distance Estimation

Consider a database consisting of private information, for example medical histories of patients, encoded as points in a space. We would like to be able to search the database for points that are similar to a given query point $\mathbf{q}$ without revealing sensitive information about the data points, except whether there exists a point within a given distance $r$ from $\mathbf{q}$. This is nontrivial even in the case of a single point, so we focus on the distance estimation problem: Is the distance between points $\mathbf{q}$ and $\mathbf{x}$ at most $r$ or not? Given $\mathbf{x}$ we would like to answer this while revealing as little information as possible about $\mathbf{x}$. Secure multi-party computations can be used for this (see e.g. (Goldreich, 2009)), but such protocols are not practical in general. In contrast, the intersection of two sets can be computed efficiently and in a private way that does not reveal anything about the items not in the intersection (Freedman et al., 2004; De Cristofaro et al., 2010; Pinkas et al., 2017).

We are going to allow false positives and approximate false negatives as follows. For an approximation factor $c>1$, and parameters $\varepsilon,\delta>0$:

• •

If $\mathbf{q}$ and $\mathbf{x}$ have distance at most $r$, we say “Yes” with probability at least $1-\varepsilon$.

• •

If $\mathbf{q}$ and $\mathbf{x}$ have distance at least $cr$, we say “No” with probability at least $1-\delta$.

Our approach is to reduce this problem to private set intersection, as follows: For a parameter $t$ to be chosen later, pick a DSH family $\mathcal{H}$ with a step-function CPF with collision probability $\Theta(1/t)$ at distances in $[0,r]$, and let $\rho>0$ be the smallest constant such that the collision probability for distances larger than $cr$ is $O(t^{-1/\rho})$. Without loss of generality we may assume that hash values are $O(\log t)$ bits (if not, hash them to this number of bits using universal hashing, increasing the collision probability only slightly). Now generate a sequence of $O(t\log(1/\varepsilon))$ hash functions pairs $(h_{1},g_{1}),(h_{2},g_{2}),\dots\sim\mathcal{H}$, independently and consider the vectors $(h_{1}(\mathbf{x}),h_{2}(\mathbf{x}),\dots)$ and $(g_{1}(\mathbf{x}),g_{2}(\mathbf{x}),\dots)$. By definition the expected (component-wise) intersection size of the two vectors, i.e., the expected number of hash collisions, is $O(\log(1/\varepsilon))$. This means that the intersection of the two vectors reveals $O(\log(1/\varepsilon)\log t)$ bits of information about the vectors in expectation. Also, by adjusting constants, the probability that the intersection size is [math] when $\mathbf{q}$ and $\mathbf{x}$ are close is at most $\varepsilon$. Thus, saying “Yes” if and only if the intersection is nonempty satisfies the first condition. On the other hand, by a union bound the probability of a “Yes” when $\mathbf{q}$ and $\mathbf{x}$ have distance at least $cr$ is $\delta=O(t\log(1/\varepsilon)/t^{1/\rho})$. Conversely, we need $t\approx(1/\delta)^{\rho/(1-\rho)}$ to have false positive probability $\delta$.

We observe that our approach has a stronger privacy constraint than the result in (Riazi et al., 2016). Indeed, the step-function DSH preserves privacy even if the $\mathbf{q}$ and $\mathbf{x}$ points are very close (e.g., $\mathbf{q}=\mathbf{x}$), since the collision probability is almost equal in the range $[0,r]$. On the other hand, a standard LSH, as the one adopted in (Riazi et al., 2016), has an high collision rate when the points are very close, revealing information on near points.

7. Conclusion

We have initiated the study of distance-sensitive hashing, an asymmetric class of hashing methods that considerably extend the capabilities of standard LSH. We proposed different constructions of such hash families and described some applications. Interestingly, DSHs provide an unique framework for capturing problems that have been separately studied, like nearest neighbor (Indyk and Motwani, 1998), finding orthogonal vectors (Vijayanarasimhan et al., 2014), furthest point query (Pagh et al., 2017), and privacy preserving search (Riazi et al., 2016).

Though we settled some basic questions regarding what is possible using DSH, many questions remain. Ultimately, one would like for a given space a complete characterization of the CPFs that can be achieved, with emphasis on extremal properties. For example: For a CPF that has $f(x)=\Theta(\varepsilon)$ for $x\in[0,r]$, how small a value $\rho_{+}(c)=\log(f(r))/\log(f(cr))$ is possible outside of this range? Additionally, our solution to the annulus problem works by combining an LSH and an anti-LSH family to obtain a unimodal family: While we know lower bounds for both, it is not clear whether combining them yields optimal solutions for this problem. Finally, it is also of interest to consider other applications of DSH in privacy preserving search and in kernel density estimation (e.g. (Lambert et al., 1999)).

Acknowledgements.

The authors thank Thomas D. Ahle for insightful conversations. The research leading to these results has received funding from the Sponsor European Research Council under the European Union’s 7th Framework Programme (FP7/2007-2013) / ERC grant agreement no. Grant #614331. Silvestri has also been supported by project Grant #SID2017 of the Sponsor University of Padova . BARC, Basic Algorithms Research Copenhagen, is supported by the Sponsor VILLUM Foundation grant Grant #16582.

Appendix A Monotone DSH constructions

A.1. Optimal monotone DSH for the unit sphere

We initially bound the CPF of the family $\mathcal{D}_{+}$, which translates to a bound for $\mathcal{D}_{-}$ through the following observation:

Lemma A.1.

Given $\mathcal{D}_{+}$ and $\mathcal{D}_{-}$ with identical parameters, we have that $f_{+}(\alpha)=f_{-}(-\alpha)$.

Proof.

A bivariate normally distributed variable with correlation $\alpha$ can be represented as a pair $(X,Y)$ with $X=Z_{1}$ and $Y=\alpha Z_{1}+\sqrt{1-\alpha^{2}}Z_{2}$ where $Z_{1},Z_{2}$ are i.i.d. standard normal. By the symmetry of the standard normal distribution around zero it is straightforward to verify that $\Pr[Z_{1}\geq t\land\alpha Z_{1}+\sqrt{1-\alpha^{2}}Z_{2}\geq t]=\Pr[Z_{1}\geq t\land-\alpha Z_{1}+\sqrt{1-\alpha^{2}}Z_{2}\leq-t]$. ∎

The collision probability for $(h,g)\sim\mathcal{D}_{+}$ depends only on the inner product $\alpha=\langle{\mathbf{x}},{\mathbf{y}}\rangle$ between the pair of points being evaluated and is given by

[TABLE]

To see why this is the case first note that it is only possible that $h(\mathbf{x})=g(\mathbf{y})$ in the event that $h(\mathbf{x})\leq m\lor g(\mathbf{y})\leq m$. Conditioned on this happening, consider the first $i$ such that either $\langle{\mathbf{z}_{i}},{\mathbf{x}}\rangle\geq t$ or $\langle{\mathbf{z}_{i}},{\mathbf{y}}\rangle\geq t$. Now the probability of collision is given by $\Pr[\langle{\mathbf{z}},{\mathbf{x}}\rangle\geq t\land\langle{\mathbf{z}},{\mathbf{y}}\rangle\geq t|\langle{\mathbf{z}},{\mathbf{x}}\rangle\geq t\lor\langle{\mathbf{z}},{\mathbf{y}}\rangle\geq t]$.

To bound the CPF of $\mathcal{D}_{+}$ and $\mathcal{D}_{-}$, we use the following tail bounds for the standard normal distribution and the tail bounds by Savage (Savage, 1962) for the bivariate standard normal distribution.

Lemma A.2 (Follows Szarek & Werner (Szarek and Werner, 1999)).

Let $Z$ be a standard normal random variable. Then, for every $t\geq 0$ we have that

[TABLE]

Lemma A.3 (Savage (Savage, 1962)).

Let $\alpha\in(-1,1)$ and let $Z_{1},Z_{2}\sim\mathcal{N}(0,1)$. Define $X_{1}=Z_{1}$ and $X_{2}=\alpha Z_{1}+\sqrt{1-\alpha^{2}}Z_{2}$. Then, for every $t>0$ we have that

[TABLE]

Corollary A.4.

By symmetry of the normal distribution the Lemma A.3 bounds apply to $\Pr[X_{1}\geq t\land X_{2}\leq-t]$ when we replace all occurrences of $\alpha$ with $-\alpha$.

We are now ready to bound the CPF for $\mathcal{D}_{+}$.

Lemma A.5.

For every $t>0$ and $\alpha\in(-1,1)$ the family $\mathcal{D}_{+}$ satisfies

[TABLE]

The complexity of sampling, storing, and evaluating a pair of functions $(h,g)\in\mathcal{D}_{+}$ is $O(dt^{4}e^{t^{2}/2})$.

Proof.

We proceed by deriving upper and lower bounds on the collision probability.

[TABLE]

We derive the lower bound in stages.

[TABLE]

The first part is lower bounded by

[TABLE]

The probability of not being captured by a projection depends on the number of projections $m$. In order to make this probability negligible we can set $m=\lceil 2t^{3}/p^{\prime}\rceil$ where $p^{\prime}$ denotes the lower bound from Lemma A.2.

[TABLE]

The bound on the complexity of sampling, storing, and evaluating a pair of functions $(h,g)\in\mathcal{D}_{+}$ follows from having $m=\lceil 2t^{3}/p^{\prime}\rceil=O(t^{4}e^{t^{2}/2})$ standard normal projections of length $d$ to be sampled, stored, and evaluated. ∎

Combining the above ingredients we get the following results, which implies Theorem 1.2 by Lemma A.1.

Theorem A.6.

For every $t>1$ there exists a distance-sensitive family $\mathcal{D}_{+}$ for $(\mathbb{S}^{d-1},\langle{\cdot},{\cdot}\rangle)$ with a CPF $f$ such that for every $\alpha\in(-1,1)$ satisfying $|\alpha|<1-1/t$ we have that

[TABLE]

Furthermore, the CPF of $\mathcal{D}_{+}$ is monotonically increasing, and the complexity of sampling, storing, and evaluating $(h,g)\in\mathcal{D}_{-}$ is $O(dt^{4}e^{t^{2}/2})$.

A more careful analysis of the collision probabilities is required in order to combine the families $\mathcal{D}_{-}$ and $\mathcal{D}_{+}$ to form a unimodal family that can be used to solve the annulus search problem, see Theorem 6.1. These results are stated in Appendix 6.2.

Appendix B DSH for Euclidean space

Proof of Theorem 4.1.

For the sake of simplicity we assume $r=1$ in the analysis (otherwise it is enough to scale down vectors accordingly). Let $\mathbf{x}$ and $\mathbf{y}$ be two points in $\mathbb{R}^{d}$ with distance $\Delta$. We know that for $\mathbf{a}\sim\mathcal{N}^{d}(0,1)$ the inner product $\langle\mathbf{a},(\mathbf{x}-\mathbf{y})\rangle$ is distributed as $\mathcal{N}(0,\Delta)$. A necessary but not sufficient condition to have a collision between $\mathbf{x}$ and $\mathbf{y}$ is that $\langle\mathbf{a},(\mathbf{x}-\mathbf{y})\rangle$ lies in the interval $[(k-1)w,(k+1)w]$. Now, if $t:=\langle\mathbf{a},(\mathbf{x}-\mathbf{y})\rangle\in[(k-1)w,kw]$, then the random offset $b$ must lie in an interval of length $t-(k-1)w$, putting $\langle\mathbf{a},\mathbf{x}\rangle$ and $\langle\mathbf{a},\mathbf{y}\rangle-(k-1)w$ into different buckets. For the interval $[kw,(k+1)w]$ similar observations show that $b$ has to be chosen in an interval of length $(k+1)w-t$. Let $\phi(t)=1/\sqrt{2\pi}e^{-t^{2}/2}$ be the density function of a standard normal random variable. Similarly to the calculations in (Datar et al., 2004), the collision probability at distance $\Delta$ can be calculated as follows:

[TABLE]

We now proceed to upper bound $\rho^{-}$ by finding an upper bound on $f(1/c)$ and a lower bound on $f(1)$. Simple calculations give an upper bound of

[TABLE]

For the lower bound, we only look at the interval $t\in[kw,(k+1/2)w]$ and obtain the bound:

[TABLE]

Now we multiply the ratio of the logarithms of the right-hand sides of (5) and (6) with $c^{2}$ and proceed to show that this term is bounded by $1+O(1/k)$, which shows the result. In the following, we set $w$ such that $w\leq\sqrt{2\pi}/(2c)$. We compute:

[TABLE]

∎

Appendix C General constructions

C.1. Proof of Lemma 1.4

Proof.

We consider the transformation in (Chierichetti and Kumar, 2015) in the asymmetric setting. Let $\mathbf{x},\mathbf{y}$ be two arbitrary points from $X$. Part (a): Sample a pair $(h_{i},g_{i})$ from $\mathcal{D}_{i}$ for each $i\in\{1,\ldots,n\}$ and set $h(\mathbf{x})=(h_{1}(\mathbf{x}),\ldots,h_{n}(\mathbf{x}))$ and $g(\mathbf{y})=(g_{1}(\mathbf{y}),\ldots,g_{n}(\mathbf{y}))$. We observe that

[TABLE]

Part (b): Pick an integer $i\in\{1,\ldots,n\}$ according to $\{p_{i}\}$ at random. Then sample a pair $(h_{i},g_{i})$ from $\mathcal{D}_{i}$. The hash function pair $(h,g)$ is given by $(i,h_{i}(\mathbf{x}))$ and $(i,g_{i}(\mathbf{y}))$. We observe that

[TABLE]

∎

C.2. Angular similarity function

This section shows how to derive a distance sensitive scheme with collision probability $\text{sim}(\mathcal{P}\left(\langle{\mathbf{x}},{\mathbf{y}}\rangle\right))$, when $\sum_{i=0}^{k}|a_{i}|=1$. Figure 4 gives some examples of functions that can be obtained from Theorem 5.1 using SimHash (Charikar, 2002).

Proof of Theorem 5.1.

Valiant (Valiant, 2015) has shown how, for any real degree-$k$ polynomial $p$, to construct a pair of mappings $\varphi^{p}_{1},\varphi^{p}_{2}:{\mathbb{R}}^{d}\rightarrow{\mathbb{R}}^{D}$, where $D=O(d^{k})$, such that $\varphi^{p}_{1}(\mathbf{x})\cdot\varphi^{p}_{2}(\mathbf{y})=\mathcal{P}\left(\langle{\mathbf{x}},{\mathbf{y}}\rangle\right)$. For completeness we outline the argument here: First consider the monomial $\mathcal{P}\left(t\right)=a_{k}t^{k}$. For $\mathbf{x}\in{\mathbb{R}}^{d}$, let $\mathbf{x}^{(k)}$ denote the vector of dimension $d^{k}$ indexed by vectors ${\bf i}=(i_{1},\dots,i_{k})\in[d]^{k}$, where $\mathbf{x}^{(k)}_{\bf i}=\prod_{j=1}^{k}x_{i_{j}}$. It is easy to verify that $\langle{\mathbf{x}^{(k)}},{\mathbf{y}^{(k)}}\rangle=(\langle{\mathbf{x}},{\mathbf{y}}\rangle)^{k}$ for all $\mathbf{x},\mathbf{y}\in{\mathbb{R}}^{d}$. With this notation in place we can define $\varphi^{p}_{1}(\mathbf{x})=\sqrt{|a_{i}|}\,\mathbf{x}^{(k)}$ and $\varphi^{p}_{2}(\mathbf{y})=(a_{i}/\sqrt{|a_{i}|})\,\mathbf{y}^{(k)}$ which satisfy $\varphi^{p}_{1}(\mathbf{x})\cdot\varphi^{p}_{2}(\mathbf{x})=a_{i}(\langle{\mathbf{x},\mathbf{y}},{)}\rangle^{k}$. The asymmetry of the mapping is essential to allow a negative coefficient $a_{k}$. To handle an arbitrary real polynomial $\mathcal{P}\left(t\right)=\sum_{i=0}^{k}a_{i}t^{i}$ we simply concatenate vectors corresponding to each monomial, obtaining a vector of dimension $\sum_{i=0}^{k}d^{i}=O(d^{k})$.

Observe that $||\mathbf{x}^{(k)}||_{2}^{2}=\langle{\mathbf{x}^{(k)}},{\mathbf{x}^{(k)}}\rangle=(\langle{\mathbf{x},\mathbf{x}},{)}\rangle^{k}=||\mathbf{x}||_{2}^{2k}$. This means that for $||\mathbf{x}||_{2}^{2}=1$ we have $||\varphi^{p}_{1}(\mathbf{x})||^{2}=\sum_{i=0}^{k}\sqrt{|a_{i}|}^{2}=1$, using the assumption $\sum_{i=0}^{k}|a_{i}|=1$. Similarly, we have for $||\mathbf{y}||_{2}^{2}=1$ that $||\varphi^{p}_{2}(\mathbf{y})||^{2}=\sum_{i=0}^{k}(a_{i}/\sqrt{|a_{i}|})^{2}=\sum_{i=0}^{k}|a_{i}|=1$. Thus, $\varphi^{p}_{1}$ and $\varphi^{p}_{2}$ map $S^{d-1}$ to $S^{D-1}$.

Our family $\mathcal{F}$ samples a function $s$ from the distribution $\mathcal{S}$ corresponding to sim and constructs the function pair $(h,g)$ with $h(\mathbf{x})=s(\varphi^{p}_{1}(\mathbf{x}))$, $g(\mathbf{y})=s(\varphi^{p}_{2}(\mathbf{y}))$. Using the properties of the functions involved we have

[TABLE]

∎

C.3. Hamming distance functions

Proof of Theorem 5.2.

We initially assume that $a_{0}\neq 0$ (i.e., 0 is not a root of $\mathcal{P}\left(t\right)$), and then remove this assumption at the end of the proof. We recall that a root of $\mathcal{P}\left(t\right)$ can appear with multiplicity larger than 1 and that, by the complex conjugate root theorem, if $z=a+bi$ is a complex root then so is its conjugate $z^{\prime}=a-bi$. We let $Z$ be the multiset containing the $k$ roots of $\mathcal{P}\left(t\right)$, with $Z_{r+}$ and $Z_{r-}$ being the multiset of positive and negative real roots, respectively, and with $Z_{c}$ being the multiset consisting of pairs of conjugate complex roots. By factoring $\mathcal{P}\left(t\right)$, we get:

[TABLE]

where the last step follows since $a_{k}\prod_{z\in Z_{r+}}(z-t)=|a_{k}|\prod_{z\in Z_{r+}}(t-z)>0$. Indeed, $\mathcal{P}\left(t\right)$ is positive in $(0,1)$ and the multiplicative terms associated with complex and negative real roots are positive in this range; this implies that the remaining terms are positive as well.

We need to introduce scaled and biased variations of bit-sampling or anti bit-sampling. Anti-bit sampling with scaling factor $\alpha\in[0,1]$ and bias $\beta\in[0,1]$ has the CPF $f(t)=\beta/2+\alpha t/2$ and is given by randomly selecting one of following two schemes: (1) with probability $1/2$, the scheme is a standard hashing that maps data and query points to 0 with probability $\beta$, and otherwise to 0 and 1 respectively; (2) with probability $1/2$, the scheme is anti bit-sampling where the sampled bit is set to 0 with probability $1-\alpha$ on both data and query points, or kept unchanged otherwise. Similarly, bit-sampling with scaling factor $\alpha\in[0,1]$ has the CPF $f(t)=(1-\alpha t)$ and is given by using bit-sampling, where the sampled bit is set to 0 with probability $1-\alpha$ on both data and query points. (We do not need a biased version of bit-sampling.)

We now assign to each multiplicative term of (C.3) a scaled and biased version of bit-sampling or anti bit-sampling as follows:

• •

$z$** is real and $z<-1$.** We assign to $z$ an anti bit-sampling with bias $1$ and scaling factor $1/|z|\leq 1$: the CPF is $S_{1}(t,z)=(1/2+t/(2|z|))$, and we have $(t+|z|)=2|z|S_{1}(t,z)$.

• •

$z$** is real and $-1\leq z<0$.** We assign to $z$ an anti bit-sampling with bias $|z|\leq 1$ and scaling factor $1$: the CPF is $S_{2}(t,z)=|z|/2+t/2$, and we have $(t+|z|)=2S_{2}(t,z)$.

• •

$z$** is real and $z\geq 1$.** We assign to $z$ a bit-sampling with scaling factor $1/z\leq 1$: the CPF is $S_{3}(t,z)=(1-t/z)$, and we have $(t-z)=zS_{3}(t,z)$.

• •

$(z,z^{\prime})$** are conjugate complex roots and $\textnormal{Real}(z)<-1$.** Let $z=a+bi$ and $z^{\prime}=a-bi$. The assigned scheme has CPF

[TABLE]

and is obtained as follows: with probability $b^{2}/(a^{2}+b^{2})$, the scheme maps data and query points to 0 and 0 with probability 1/4, or to 0 and 1 with probability 3/4; with probability $a^{2}/(a^{2}+b^{2})$, the schemes consists of the concatenation of two anti bit-sampling with bias $1$ and scaling factor $1/|a|$. Note that $t^{2}-2at+a^{2}+b^{2}=4(a^{2}+b^{2})S_{4}(t,z)$.

• •

$(z,z^{\prime})$** are conjugate complex roots and $\textnormal{Real}(z)\geq 1$.** The scheme is similar to the previous one where we use two bit-sampling with scaling factor $1/a$ instead of the anti bit-sampling. The CPF is

[TABLE]

and we get $t^{2}-2at+a^{2}+b^{2}=(a^{2}+b^{2})S_{5}(t,z)$.

• •

$(z,z^{\prime})$** are conjugate complex roots, $-1\leq\textnormal{Real}(z)\leq 0$, and $|z|=a^{2}+b^{2}\geq 1$.** We assign the following scheme with CPF

[TABLE]

With probability $1/4$ the scheme maps data and query points to 0; with probability $1/2$, the scheme consists of anti bit-sampling with bias 0 and scaling factor $|a|/(a^{2}+b^{2})\leq 1$; with probability $1/4$ the scheme consists of two anti bit-sampling with bias 0 and scaling factor $\sqrt{a^{2}+b^{2}}$ each. We have $t^{2}-2at+a^{2}+b^{2}=4(a^{2}+b^{2})S_{6}(t,z)$.

• •

$(z,z^{\prime})$** are conjugate complex roots, $-1\leq\textnormal{Real}(z)\leq 0$, and $|z|=a^{2}+b^{2}<1$.** We use the scheme of the previous point with different parameters, giving CPF

[TABLE]

The scheme is the following: with probability $1/4$, the scheme is a standard hashing scheme where data points are always mapped to [math] and where a query point is mapped to [math] with probability $a^{2}+b^{2}$ and to $1$ with probability $1-a^{2}+b^{2}$; with probability $1/2$, the scheme consists of anti bit-sampling with bias 0 and scaling factor $|a|\leq 1$; with probability $1/4$, the scheme consists of two anti bit-sampling with bias 0 and scaling factor $1$ each. We have $t^{2}-2at+a^{2}+b^{2}=4S_{7}(t,z)$.

Consider the scheme obtained by concatenating the above ones for each real root and each pair of conjugate roots. Its CPF is $S(t)=\prod_{i=1}^{6}\prod_{z\in Z_{i}}S_{i}(t,z)$, where $Z_{i}$ contains root with CPF $S_{i}$. Then, by letting $\psi$ denote the number of roots with negative real part, we get from Equation C.3:

[TABLE]

Consider now $a_{k}=0$ and let $\ell$ be the largest value such that $\mathcal{P}\left(t\right)=t^{\ell}\mathcal{P}^{\prime}\left(x\right)$ with $\mathcal{P}^{\prime}\left(0\right)\neq 0$. We get the claimed result by concatenating $\ell$ anti bit-sampling, which gives a CPF of $x^{\ell}$, and the scheme for $\mathcal{P}^{\prime}\left(t\right)$ obtained by the procedure described above. ∎