Fast Similarity Sketching

TL;DR

This paper introduces a new approach to similarity sketching that aims to efficiently preserve Jaccard similarity between sets with strong concentration guarantees, improving upon traditional methods like MinHash.

Contribution

The paper proposes a novel similarity sketching method that offers faster computation while maintaining strong concentration bounds for Jaccard similarity estimation.

Findings

Achieves similar accuracy to MinHash with reduced computational complexity

Provides theoretical guarantees for concentration bounds

Demonstrates improved performance in large-scale similarity search

Abstract

We consider the $\textit{Similarity Sketching}$ problem: Given a universe $[u] = \{0,\ldots, u-1\}$ we want a random function $S$ mapping subsets $A\subseteq [u]$ into vectors $S(A)$ of size $t$, such that the Jaccard similarity $J(A,B) = |A\cap B|/|A\cup B|$ between sets $A$ and $B$ is preserved. More precisely, define $X_i = [S(A)[i] = S(B)[i]]$ and $X = \sum_{i\in [t]} X_i$. We want $E[X_i]=J(A,B)$, and we want $X$ to be strongly concentrated around $E[X] = t \cdot J(A,B)$ (i.e. Chernoff-style bounds). This is a fundamental problem which has found numerous applications in data mining, large-scale classification, computer vision, similarity search, etc. via the classic MinHash algorithm. The vectors $S(A)$ are also called $\textit{sketches}$. Strong concentration is critical, for often we want to sketch many sets $B_1,\ldots,B_n$ so that we later, for a query set $A$, can find (one of) the most similar $B_i$. It is then critical that no $B_i$ looks much more similar to $A$ due to errors in the sketch. The seminal $t\times\textit{MinHash}$ algorithm uses $t$ random hash functions $h_1,\ldots, h_t$, and stores $\left ( \min_{a\in A} h_1(A),\ldots, \min_{a\in A} h_t(A) \right )$ as the sketch of $A$. The main drawback of MinHash is, however, its $O(t\cdot |A|)$ running time, and finding a sketch with similar properties and faster running time has been the subject of several papers. (continued...)

Equations

$$\mathrm{Pr}\left[X\ge J(1+\delta )\right]$$

$$\mathrm{Pr}\left[X\le J(1-\delta )\right]$$

$$S(A)[j]=\min \left\{v_i(a)\mid a\in A,i\in [2t],b_i(a)=j\right\}.$$

$$(1-1/t{)}^{|A|\cdot i}\le (1-1/t{)}^{2\cdot t\log t}\le 1/t^2.$$

$$S(A\cup B,t)[i]=\min \left(S(A,t)[i],S(B,t)[i]\right).$$

$$\text{E}\left[\prod _{i\in I}{X}_i\right]\le J^k,$$

$$\text{E}\left[\prod _{i\in I}{X}_i\right]\ge \frac{(tJ{)}^{\underline{k}}}{t^{\underline{k}}}.$$

$$p=\sum _{j\in [2t]}\frac{\left|T_j\right|-\left|T_j\cap I^{\prime }\right|}{t-i}\cdot \frac{nJ-\left|T_j\cap I^{\prime }\right|}{n-\left|T_j\cap I^{\prime }\right|}.$$

$$J\ge \frac{nJ-\left|T_j\cap I^{\prime }\right|}{n-\left|T_j\cap I^{\prime }\right|}\ge \frac{\left|T_j\right|J-\left|T_j\cap I^{\prime }\right|}{\left|T_j\right|-\left|T_j\cap I^{\prime }\right|},$$

$$J=\sum _{j\in [2t]}\frac{\left|T_j\right|-\left|T_j\cap I^{\prime }\right|}{t-i}\cdot J\ge p\ge \sum _{j\in [2t]}\frac{\left|T_j\right|-\left|T_j\cap I^{\prime }\right|}{t-i}\cdot \frac{\left|T_j\right|J-\left|T_j\cap I^{\prime }\right|}{\left|T_j\right|-\left|T_j\cap I^{\prime }\right|}=\frac{tJ-i}{t-i}.$$

$$\mathrm{Pr}\left[X\ge J(1+\delta )\right]$$

$$\mathrm{Pr}\left[X\le J(1-\delta )\right]$$

$$1-\frac{e^{-{\delta }^{2}r/2}}{1-e^{-{\delta }^2/2}}.$$

$$e^{-2{\delta }^{2}t}.$$

$$\text{E}\left[Y_i\right]\ge \frac{(Jt{)}^{\underline{K}}}{t^{\underline{K}}}>{\left(\frac{tJ-K}{t-K}\right)}^K=J^K\cdot {\left(1-\frac{K(1-J)}{J(t-K)}\right)}^K.$$

$$\text{E}\left[Y_{i_0}{Y}_{i_1}\right]\le J^{2K}\cdot {\left(1+\frac{K(1-J)}{Jt}\right)}^K.$$

$$R=\left\{T[i,j]\mid (i,j)\in \left\{i_0,i_1\right\}\times [L]\right\}.$$

$$\text{E}\left[Y_{i_0}{Y}_{i_1}\right]\le \text{E}\left[J^K\prod _{j\in [K]}{J}^{Z_j}\right]=J^K\prod _{j\in [K]}\text{E}\left[J^{Z_j}\right].$$

$$\text{E}\left[J^{Z_j}\right]=\left(1-\frac{K}{t}\right)\cdot J+\frac{K}{t}=J\cdot \left(1+\frac{K(1-J)}{Jt}\right),$$

$$\text{E}\left[Y(Y-1)\right]=\sum _{i_0,i_1\in L,i_0\ne i_1}\text{E}\left[Y_{i_0}{Y}_{i_1}\right]\le L(L-1)\cdot e\cdot J^{2K}<L^2\cdot e\cdot J^{2K}.$$

$$\mathrm{Pr}\left[Y>0\right]\ge \frac{(\text{E}\left[Y\right]{)}^2}{\text{E}\left[Y^2\right]}\ge \frac{(L\cdot J^K{)}^2/16}{e(L\cdot J^K{)}^2+(L\cdot J^K)}=\frac{1}{16(e+(L\cdot J^K{)}^{-1})}\ge \frac{1}{16(e+1)}=\Omega (1).$$

$${\tau }_F\le 3r+\sum _{i=2r+1}^t{T}_i.$$

$$\text{E}\left[T_i\right]\le i\cdot \min \left\{\mathrm{Pr}\left[X_{<i}\ge i\cdot \gamma +\sqrt[3]{i^2}\right],\mathrm{Pr}\left[X_{<j}<j\cdot \gamma +\sqrt[3]{j^2}\right]\right\}.$$

$$X_{<i}-\text{E}\left[X_{<i}\right]\ge i\cdot \gamma +\sqrt[3]{i^2}-i{\gamma }^{\prime }=\Omega \left(\sqrt[3]{i^2}\right).$$

$$\mathrm{Pr}\left[X_{<i}-\text{E}\left[X_{<i}\right]\ge i\cdot \gamma +\sqrt[3]{i^2}-i{\gamma }^{\prime }\right]\le e^{-\Omega (\sqrt[3]{i})}.$$

$$\text{E}\left[\sum _{i=2r}^t{T}_i\right]\le \sum _{i=2r}^{t}i\cdot e^{-\Omega (\sqrt[3]{i})}\le \sum _{i\ge 1}i\cdot e^{-\Omega (\sqrt[3]{i})}=O(1).$$

$$\sum _{i=r}^t\mathrm{Pr}\left[X_{<i}\le i\cdot \gamma +\sqrt[3]{i^2}\right]$$

$$\le \sum _{i=r}^t\mathrm{Pr}\left[X_{<i}-\text{E}\left[X_{<i}\right]\le -i\cdot \delta /2\right]$$

$$\le \sum _{i=r}^t{e}^{-{\delta }^{2}i/2}$$

$$\le \sum _{i\ge r}{e}^{-{\delta }^{2}i}=\frac{e^{-{\delta }^{2}r/2}}{1-e^{-{\delta }^2/2}},$$

$$\mathrm{Pr}\left[X_{<t}>t\cdot \gamma +\sqrt[3]{t^2}\right]\le \mathrm{Pr}\left[X_{<t}>t\cdot \gamma \right]\le \mathrm{Pr}\left[X_{<t}-\text{E}\left[X_{<t}\right]>t\cdot \delta \right]\le e^{-2{\delta }^{2}t},$$

Full text

The seminal $t\times$MinHash algorithm uses $t$ random hash functions $h_{1},\ldots,h_{t}$, and stores $\left(\min_{a\in A}h_{1}(A),\ldots,\min_{a\in A}h_{t}(A)\right)$ as the sketch of $A$. The main drawback of MinHash is, however, its $O(t\cdot|A|)$ running time, and finding a sketch with similar properties and faster running time has been the subject of several papers. Addressing this, Li et al. [NIPS’12] introduced one permutation hashing (OPH), which creates a sketch of size $t$ in $O(t+|A|)$ time, but with the drawback that possibly some of the $t$ entries are “empty” when $|A|=O(t)$. One could argue that sketching is not necessary in this case, however the desire in most applications is to have one sketching procedure that works for sets of all sizes. Therefore, filling out these empty entries is the subject of several follow-up papers initiated by Shrivastava and Li [ICML’14]. However, these “densification” schemes fail to provide good concentration bounds exactly in the case $|A|=O(t)$, where they are needed.

In this paper we present a new sketch which obtains essentially the best of both worlds. That is, a fast $O(t\log t+|A|)$ expected running time while getting the same strong concentration bounds as MinHash. Our new sketch can be seen as a mix between sampling with replacement and sampling without replacement. We demonstrate the power of our new sketch by considering popular applications in large-scale classification with linear SVM as introduced by Li et al. [NIPS’11] as well as approximate similarity search using the LSH framework of Indyk and Motwani [STOC’98]. In particular, for the $j_{1},j_{2}$-approximate similarity search problem on a collection of $n$ sets we obtain a data-structure with space usage $O(n^{1+\rho}+\sum_{A\in\mathcal{C}}|A|)$ and $O(n^{\rho}\log n+|Q|)$ expected time for querying a set $Q$ compared to a $O(n^{\rho}\log n\cdot|Q|)$ expected query time of the classic result of Indyk and Motwani.

1 Introduction

In this paper we consider the following problem which we call the similarity sketching problem. Given a large universe $[u]=\{0,\ldots,u-1\}$ and positive integer $t$ we want a random function $S$ mapping subsets $A\subseteq[u]$ into vectors (which we will call sketches) $S(A)$ of size $t$, such that similarity is preserved. More precisely, given sets $A,B\subseteq[u]$, define $X_{i}=[S(A)[i]=S(B)[i]]$ for each $i\in[t]$, where $S(A)[i]$ denotes the $i$th entry of the vector $S(A)$ and $[x]$ is the Iverson bracket notation with $[x]=1$ when $x$ is true and [math] otherwise. Let $X=\sum_{i\in[t]}X_{i}$, then we want $\text{E}\!\left[{X}\right]=t\cdot J(A,B)$, where $J(A,B)=|A\cap B|/|A\cup B|$ is the Jaccard similarity of $A$ and $B$. That is, the sketches can be used to estimate $J(A,B)$ by doing a pair-wise comparison of the entries. We will call this the alignment property of the similarity sketch. Finally, we want to have Chernoff-style concentration bounds on the value of $X$. The standard solution to this problem is $t\times$MinHash algorithm111https://en.wikipedia.org/wiki/MinHash. The algorithm works as follows: Let $h_{0},\ldots,h_{t-1}:[u]\to[0,1]$ be random hash functions and define $S(A)=(\min_{a\in A}h_{0}(a),\ldots,\min_{a\in A}h_{t-1}(a))$. This corresponds to sampling $t$ elements from $A$ with replacement and thus has all the above desired properties.

MinHash was originally introduced by Broder et al. [5, 6] for the AltaVista search engine and has since been used as a standard tool in many applications including duplicate detection [6, 9], all-pairs similarity [4], large-scale learning [14], computer vision [17], and similarity search [11]. The main motivation for hashing-based approaches to these problems is the continuing increases in dimensionality of modern datasets. Weinberger et al. [23] considered sets from a universe of size 16 trillion ($u\approx 10^{13}$) and Tong [22] considered sets with $u\approx 10^{9}$. Furthermore, when working with text, input is often represented by $w$-shingles (i.e. $w$ contiguous words) with $w\geq 5$. This further increases the dimension from, say roughly $10^{5}$ common english words to $u\approx 10^{5w}$.

The main drawback of MinHash is, however, the $O(t\cdot|A|)$ running time. For practical applications, where the data is ultra high dimensional, this sketch creation time is often a bottleneck. As an example, [14] suggests using $t=500$ and [12] suggests using $t=4000$. Several papers have therefore been concerned with finding a similarity sketch with equal power and faster running time.

Bachrach and Porat [3] suggested a more efficient way of maintaining $t$ MinHash values with $t$ different hash functions. They use $t$ different polynomial hash functions that are related, yet pairwise independent, so that they can systematically maintain the MinHash for all $t$ polynomials in $O(\log t)$ time per element of $A$. There are two issues with this approach: It is specialized to work with polynomials and MinHash is known to have constant bias unless the polynomials considered have super-constant degree [15], and this bias does not decay with independent repetitions. Also, because the experiments are only pairwise independent, the concentration is only limited by Chebyshev’s inequality and thus nowhere near the Chernoff bounds we want for many applications.

Another direction introduced by Li et al. [13] is one permutation hashing (OPH) which works by hashing the elements of $A$ into $t$ buckets and performing a MinHash in each bucket using the same hash function. While this procedure gives $O(t+|A|)$ sketch creation time it also may create empty buckets and thus only obtains a sketch with $t^{\prime}\leq t$ entries when $|A|=o(t\log t)$. One may argue that sketching is not even needed in this case. However, a common goal in applications of similarity sketching is to have one sketching procedure which works for all set size – eg. one data structure that works for an entire data set of different sizes in the case of approximate similarity search. It is thus very desirable that the similarity sketch works well independently of the size of the input set.

Motivated by this, several follow-up papers [20, 19, 18] have tried to give different schemes for filling out the empty entries of the OPH sketch (“densifying” the sketch). These papers all consider different ways of copying from the full entries of the sketch into the empty ones. Due to this approach, however, these densification schemes all fail to give good concentration guarantees when $|A|$ is small, which is exactly the cases in which OPH gives many empty bins and densification is needed. This is because of the fundamental problem that unfortunate collisions in the first round cannot be resolved in the second round when copying from the full bins. To understand this consider the following extreme example: Let $A$ be a set with two elements. Then with probability $1/t$ these two elements end in the same bin, and after copying the entire densified sketch ends up consisting of just one element. This leads to very poor similarity estimation. This behaviour is illustrated with experiments in Figure 1. Furthermore, the state-of-the-art densification scheme of Shrivastava [18] has a running time of $O(t^{2})$ when $|A|=O(t)$.

1.1 Our contribution

In this paper we obtain a sketch which essentially obtains the best of both worlds. That is, strong concentration guarantees for similarity estimation as well as a fast expected sketch creation time of $O(t\log t+|A|)$. Our new sketch can be seen as a mixture between sampling with and without replacement and in many cases outperforms MinHash. An example of this can be seen in the toy example of Figure 1, where the “without replacement”-part of our sketch gives better concentration compared to MinHash. Our sketch can be directly employed in any place where $t\times$MinHash is currently employed to improve the running time. In this paper we focus on two popular applications, which are large-scale learning with linear SVM and approximate similarity search with LSH. We describe these applications in more detail below.

Another strength of our new sketch is that it can be implemented using just one mixed tabulation hash function (introduced by Dahlgaard et al. [8]) which can be evaluated in $O(1)$ time.

Theorem 1**.**

Let $[u]=\left\{0,1,2,\ldots,u-1\right\}$ be a set of keys and let $t$ be a positive integer. There exists an algorithm that given a set $A\subseteq[u]$ in expected time $O\left(\left|A\right|+t\log t\right)$ creates a size-$t$ vector $v(A)$ of non-negative real numbers with the following properties. For two sets $A,B\subseteq[u]$ it holds that $v(A\cup B)_{i}=\min\left\{v(A)_{i},v(B)_{i}\right\}$ for each index $i\in[t]$. For $i\in[t]$ let $X_{i}=1$ if $v(A)_{i}=v(B)_{i}$ and [math] otherwise and let $X=\frac{1}{t}\sum_{i\in[t]}X_{i}$. Then $\text{E}\!\left[{X}\right]=J$ where $J=J(A,B)$ and for $\delta>0$ it holds that:

[TABLE]

Large-scale learning

Li et al. [14] considered using similarity sketching for applications in large-scale linear learning. In particular they showed how to naturally integrate MinHash into linear SVM and logistic regression to avoid computations on extremely large data points. The idea is as follows: For each input set $A$, they create a $t\times$MinHash similarity sketch and truncate each value in the sketch to $b$ bits (called $b$-bit minwise hashing). They then create a vector of size $2^{b}\cdot t$ by concatenating the indicator vectors (of size $2^{b}$) for each entry in the $b$-bit similarity sketch. By the alignment property of the similarity sketch it follows that the Jaccard similarity of two sets can be estimated as the dot-product of the two corresponding size-$2^{b}t$ vectors (with a bias depending on $b$). This is exactly the property needed by a linear SVM in order to perform efficient classification. As the linear SVM performs classification using a single dot-product the classification time then becomes $O(2^{b}t+t\cdot|A|)$ when using $t\times$MinHash. Using our new similarity sketch we immediately improve this to $O((2^{b}+\log t)\cdot t+|A|)$ which removes a major bottleneck (see [13]).

We note that it is crucial to this application that the similarity sketch satisfies the alignment property as also noted by Li et al. [13], as the similarity estimation can otherwise not be implemented with a dot-product.

Speeding up LSH

One of the most popular applications of the MinHash algorithm is the approximate similarity search problem. Here, we are given a collection, $\mathcal{C}$, of $n$ sets from some universe $[u]$ as well as two parameters $0\leq j_{2}<j_{1}\leq 1$. The task is to pre-process $\mathcal{C}$ such that given a query set $Q\subseteq[u]$ we can efficiently return a set $A\in\mathcal{C}$ with $J(A,Q)\geq j_{2}$ if there exists some $B\in\mathcal{C}$ with $J(B,Q)\geq j_{1}$. It is common to assume that $j_{1},j_{2}$ are constants and we do the same in this paper. To address this problem, Indyk and Motwani [11] introduced the Locality Sensitive Hashing (LSH) framework. For parameters $L,K$, they created $L$ different $K\times$MinHash sketches, $S_{0}(A),\ldots,S_{L-1}(A)$ for each set $A\in\mathcal{C}$. A query is then answered by computing $L$ different $K\times$MinHash sketches $S_{0}(Q),\ldots,S_{L-1}(Q)$ for $Q$ and for each $i\in[L]$ comparing $Q$ to each set $A\in\mathcal{C}$ with $S_{i}(A)=S_{i}(Q)$. This gives a total space usage of $O(L\cdot n+\sum_{A\in\mathcal{C}}|A|)$ and an expected query time of $O(L\cdot K\cdot|Q|)$. By carefully choosing $L$ and $K$ they obtain a space usage of $O(n^{1+\rho}+\sum_{A\in\mathcal{C}}|A|)$ and expected query time of $O(|Q|\cdot n^{\rho}\log n)$, where $\rho=\log(1/j_{1})/\log(1/j_{2})$.

Following this seminal work it has become practice to evaluate algorithms in terms of their $\rho$-value, and several papers are concerned with reducing this value (see e.g. [1, 2, 7]) using increasingly sophisticated methods based on eg. data-dependant hashing as in [1, 2]. Using the LSH framework of [11] the query time is dominated by two parts: 1) The data structure returns $O(L)$ expected “false positives” which have to be filtered out in roughly $O(L\cdot|Q|)$ time, and 2) we have to compute $O(L\cdot K)$ hash values for the similarity sketches giving $O(L\cdot K\cdot|Q|)$ time when using MinHash. One way to remove this multiplicative dependance on $Q$ is by using an “intermediate” similarity sketch of size $O(\log^{3}n)$ and generating the similarity sketches of the LSH structure by sampling directly from this vector. This gives an expected query time of $O((L+|Q|)\cdot\log^{3}n)$. This is still very time consuming when $|Q|$ and $n$ are large, and thus removing the multiplicative dependance on $|Q|$ without introducing a large polylogarithmic factor was the main motivation behind studying OPH densification schemes [19, 20, 18]. However, as mentioned earlier, these densification schemes do not give the concentration bounds necessary for the LSH analysis to work.

In this paper we address the above issue, speeding up the query time of the LSH framework. Building upon ideas from Henzinger and Thorup [10] and using the similarity sketch from this paper we give a method that filters out false positives of 1) above in expected constant time, however, the main work lies in dealing with 2). To improve this part we show that we can use our new similarity sketch as an intermediate vector and sample from this in a clever way to build the LSH table in $O(L\cdot K+|Q|)$ time thus improving the total query time to $O(L\cdot K+|Q|)=O(n^{\rho}\log n+|Q|)$ expected time.

1.2 Related work

An alternative to $t\times$MinHash sketching is the bottom-$t$ sketches described in [5, 21]. The idea is to use the $t$ smallest hash values of a set $A$ instead, applying just one hash function. However, similar to OPH this does not give us a sketch of size $t$ when $|A|$ is small. Furthermore, the estimation procedure becomes more complicated and the sketches do not satisfy the alignment property, which is necessary for many applications (see Section 1.1 above).

A recent advance in approximate similarity search by Christiani and Pagh [7] is the ChosenPath method, which obtains a $\rho$-value better than the one obtained with LSH and MinHash. However, the authors consider a different similarity measure called Braun-Blanquet similarity, and in order to obtain their result for Jaccard similarity they have to convert between the two. They therefore assume that the input sets all have the same size, $\ell$, and the authors suggest222Personal communication using an $\ell\times$MinHash sketch as preprocessing to obtain this. When $\ell$ is large this pre-processing step is a bottleneck that can be sped up with out new similarity sketch.

1.3 Notation

For a real number $x$ and an integer $k$ we define $x^{\underline{k}}=x(x-1)(x-2)\ldots(x-k+1)$. For an expression $P$ we let $[P]$ denote the variable that is $1$ if $P$ is true and [math] otherwise. For a non-negative integer $n$ we let $[n]$ denote the set $[n]=\left\{0,1,2,\ldots,n-1\right\}$.

2 Fast Similarity Sketching

In this section we present our new sketching algorithm, which takes a set $A\subseteq[u]$ as input and produces a sketch $S(A,t)$ of size $t$. When $t$ is clear from the context we may write just $S(A)$.

Our new similarity sketch is simple to describe: Let $h_{0},\ldots,h_{2t-1}$ be random hash functions such that for $i\in[t]$ we have $h_{i}:[u]\to[t]\times[i,i+1)$ and for $i\in\{t,\ldots,2t-1\}$ we have $h_{i}:[u]\to\{i-t\}\times[i,i+1)$. For each hash function $h_{i}$ we say that the output is split into a bin, $b_{i}$, and a value, $v_{i}$. That is, for $i\in[2t]$ and $a\in[u]$ we have $h_{i}(a)=(b_{i}(a),v_{i}(a))$, where $b_{i}(a)$ and $v_{i}(a)$ are restricted as described above. We may then define the $j$th entry of the sketch $S(A)$ as follows:

[TABLE]

In particular, the hash functions $h_{t},\ldots,h_{2t-1}$ ensure that each entry of $S(A)$ is well-defined. Furthermore, since we have $v_{i}(a)<v_{j}(b)$ for any $a,b\in[u]$ and $0\leq i<j<2t$ we can efficiently implement the sketch defined in (1) using the procedure in Algorithm 1.

We will start our analysis of $S(A)$ by bounding the running time of Algorithm 1.

Lemma 1**.**

Let $A\subseteq[u]$ be some set and let $t$ be a positive integer. Then the expected running time of Algorithm 1 is $O(t\log t+|A|)$.

Proof.

We split the proof into two cases:

If $|A|\leq 2\log t$ we have a trivial upper bound of $O(t\cdot|A|)=O(t\log t)$. 2. 2.

Otherwise, $|A|>2\log t$. Fix $i\in[t]$ to be the smallest value in such that $|A|\cdot i>2\cdot t\log t$. Then the probability of a given bin being empty after evaluating $h_{0},\ldots,h_{i-1}$ is at most

[TABLE]

It follows that the probability of any bin being empty is at most $1/t$ and thus the expected running time is $O(|A|\cdot i+\frac{|A|\cdot t}{t})=O(t\log t+|A|)$.

∎

Next, we will prove several properties of the sketch. The first is an observation that the sketch of the union of two sets can be computed solely from the sketches of the two sets.

Fact 1**.**

Let $A,B$ be two sets and let $t$ be a positive integer. Then

[TABLE]

The main technical lemma regarding the sketch is Lemma 2 below. Loosely speaking, the lemma bounds the $k$th moment of the sketch when estimating set similarity. We will use this lemma to show that we get an unbiased estimator as well as Chernoff-style concentration bounds.

Lemma 2**.**

Let $A,B$ be sets with Jacard similarity $J(A,B)=J$ and let $t$ be a positive integer. For each $i\in[t]$ let $X_{i}=\left[S(A,t)[i]=S(B,t)[i]\right]$. Let $I\subseteq[t]$ be a set of $k$ indices. Then:

[TABLE]

and if $tJ\geq k-1$ then:

[TABLE]

Proof.

Define $\mathcal{T}=(T_{0},T_{1},\ldots,T_{2t-1})$ in the following way. Let $T_{0}=h_{0}\left(A\cup B\right)$ and for $i\geq 1$ let $T_{i}=h_{i}\left(A\cup B\right)\setminus\left(T_{0}\cup\ldots\cup T_{i-1}\right)$. Assume in the following that $\mathcal{T}$ is fixed. It clearly suffices to prove this theorem for all possible choices of $\mathcal{T}$. Let $n=\left|A\cup B\right|$, then $nJ=\left|A\cap B\right|$.

We will prove the claim when the set $I$ is chosen uniformly at random among the subsets of $[t]$ of size $k$. Because of symmetry this will suffice. More specifically let $I=\left\{v_{0},v_{1},\ldots,v_{k-1}\right\}$ where $v_{i}$ is chosen uniformly at random from $[t]\setminus\left\{v_{0},v_{1},\ldots,v_{i-1}\right\}$.

Let $i\in[k]$. Fix $v_{0},v_{1},\ldots,v_{i-1}$ and assume that $X_{v_{0}}=\ldots=X_{v_{i-1}}=1$. Let $p$ be the probability that $X_{v_{i}}=1$ conditioned on these assumptions. We will estimate $p$. Let $I^{\prime}=\left\{v_{0},\ldots,v_{i-1}\right\}$. The probability that $v_{i}\in T_{j}$ is then $\frac{\left|T_{j}\right|-\left|T_{j}\cap I^{\prime}\right|}{t-i}$. Conditioned on $v_{i}\in T_{j}$ the probability that $X_{v_{i}}=1$ is exactly $\frac{nJ-\left|T_{j}\cap I^{\prime}\right|}{n-\left|T_{j}\cap I^{\prime}\right|}$. So the probability that $X_{v_{i}}=1$ is:

[TABLE]

We note that

[TABLE]

and inserting these estimates gives that:

[TABLE]

So conditioned on $X_{v_{0}}=\ldots=X_{v_{i-1}}=1$ we conclude that the probability that $X_{v_{i}}=1$ is between $J$ and $\frac{tJ-i}{t-i}$. This implies that that the expected value of $\prod_{i\in[k]}X_{v_{i}}$ is at most $J^{k}$ and at least $\frac{(tJ)^{\underline{k}}}{t^{\underline{k}}}$ where the lower bound holds if all terms in the product are non-negative, i.e. if $tJ\geq k-1$. ∎

As a corollary we immediately get that the estimator is unbiased.

Lemma 3**.**

Let $A,B$ be sets with Jaccard similarity $J(A,B)=J$ and let $t$ be a positive integer. Let $X_{i}=[S(A,t)[i]=S(B,t)[i]]$ and let $X=\sum_{i\in[t]}X_{i}$. Then $\text{E}\!\left[{X}\right]=tJ$.

Proof.

This follows directly by applying Lemma 2 with $k=1$. ∎

We also get Chernoff-style concentration bounds as follows.

Lemma 4**.**

Let $A,B$ be sets with Jaccard similarity $J(A,B)=J$ and let $t$ be a positive integer. Let $X_{i}=[S(A,t)[i]=S(B,t)[i]]$ and let $X=\sum_{i\in[t]}X_{i}$. Then for $\delta>0$

[TABLE]

Proof.

The upper bound follows from Lemma 2 and [16, Corollary 1] since Chernoff bounds are derived by bounding $\text{E}\!\left[{e^{\lambda X}}\right]$ for some $\lambda>0$.

The lower bounds follows from considering $Y=\sum_{i\in[t]}Y_{i}$ where $Y_{i}=1-X_{i}$ and $Y=t-X$. Since $Y_{i}=[S(A\cup B,t)[i]=S((A\cup B)\setminus(A\cap B),t)[i]]$ we can use the same argument as for the upper bound, see [16, Page 4]. ∎

Practical implementation

In Algorithm 1 we used $2t$ hash functions to implement our new similarity sketch. We now briefly describe how to avoid this requirement by instead using just one Mixed Tabulation hash function as introduced by Dahlgaard et al. [8]. We do not present the entire details, but refer instead to the theorems of [8] which can be used directly in a black-box fashion.

In tabulation-based hashing we view each key, $x\in[u]$, as a vector $(x_{0},\ldots,x_{c-1})$ of $c$ characters, where each $x_{i}\in[u^{1/c}]$, and $u^{1/c}$ is called the alphabet size. Consider now the following change to Algorithm 1: Let $h$ be a mixed tabulation function with alphabet size at least $\delta\cdot t\log n$ for some sufficiently large constant $\delta$, and change Algorithm 1 to be $b,v\leftarrow h(i,a_{0},\ldots,a_{c-1})$ instead. We now consider two cases:

• •

If $|A|\leq(\delta-1)\cdot t\log n$ it follows from [8, Theorem 1], that the keys of $\{0,\ldots,i\}\times A$ all hash independently, where $i$ is an integer chosen similarly as in Lemma 1, and both correctness and running time follows immediately from the lemmas above.

• •

If $|A|>(\delta-1)\cdot t\log n$ all bins are filled out using $i=0$. In this case both correctness and running time follows immediately from [8, Theorem 2].

2.1 Separation

It can be useful to check if the Jaccard similarity of two sets are above a certain threshold or not, without having to actually calculate the Jaccard similarity. Specifically, we assume that we are given two sets $A$ and $B$ and want to determine if $J(A,B)\geq\gamma$. Intuitively, this should be easy if $J(A,B)$ is either much larger or much smaller than $\gamma$ and difficult when $J(A,B)\approx\gamma$. Inspired by Henzinger and Thorup [10] we consider the following algorithm for doing so: We let $t\geq r$ be positive integers and let $X_{i}=[S(A,t)[i]=S(B,t)[i]]$ for $i\in[t]$. We now run a for loop with an index $i$ going from $r$ to $t$. At each step we check if $\sum_{j<i}X_{j}\leq i\cdot\gamma+\sqrt[3]{i^{2}}$. If so the algorithm terminates and returns false. If no such $i$ is found the algorithm returns true. See Algorithm 2 for pseudo-code.

Assume that we use Algorithm 2 with $X_{i}=[S(A,t)[i]=S(B,t)[i]]$. In Lemma 5 we show how the algorithm behaves when $J(A,B)\geq\gamma+\delta$ and $J(A,B)\leq\gamma-\delta$ respectively. Furthermore, if we only count the running time of the algorithm in case the algorithm returns false the expected used time is $O(r)$. If $J(A,B)\geq\gamma+\delta$, $\delta$ is a constant and $r$ is a sufficiently large constant (depending on $\delta$) then the algorithm returns true with constant probability. If $J(A,B)\leq\gamma-\delta$ and $\delta$ is a constant then the algorithm returns true with probability exponentially small in $t$.

Lemma 5**.**

Let $t\geq r$ be integers and $\gamma,\delta,p\in[0,1]$. Let $X_{0},X_{1},\ldots,X_{t-1}$ be independent variables with values in $[0,1]$ such that $\text{E}\!\left[{X_{i}}\right]=p$ for every $i\in[t]$. Assume that we run Algorithm 2 with parameters $(t,(X_{0},\ldots,X_{t-1}),\gamma,r)$.

Let $\tau$ be the number of iterations of the for loop during the algorithm, and let $\tau_{F}=\tau$ if the algorithm returns false and let $\tau_{F}=0$ otherwise. Then $\text{E}\!\left[{\tau_{F}}\right]=O(r)$.

If $p\geq\gamma+\delta$ and $r\geq\frac{8}{\delta^{3}}$ the algorithm returns true with probability at least

[TABLE]

If $p\leq\gamma-\delta$ the algorithm returns true with probability at most

[TABLE]

Proof.

See Appendix A. ∎

3 Speeding up LSH

We consider the approximate similarity search problem with parameters $0<j_{2}<j_{1}<1$ on a collection, $\mathcal{C}$, of $n$ sets from a large universe $[u]$. We will create a data-structure similar to the LSH structure as described in Section 1.1 with parameters $L$ and $K$. That is, for each set $A\in\mathcal{C}$ (and query $Q$) we will create $L$ sketches $S_{0}(A),\ldots,S_{L-1}(A)$ of size $K$ such that for any two sets $A,B\subseteq[u]$ and $i\in[L]$ we have the following property:

• •

$\Pr\!\left[{S_{i}(A)=S_{i}(B)}\right]\leq J(A,B)^{K}$.

• •

If $J(A,B)\geq j_{1}$ then $\Pr\!\left[{S_{i}(A)=S_{i}(B)}\right]=\Theta(J(A,B)^{K})$.

By setting $K=\left\lceil{\frac{\log n}{\log(1/j_{2})}}\right\rceil$ and $L=\left\lceil{(1/j_{1})^{K}}\right\rceil$ and using the analysis of [11] this immediately gives us $O(n^{1+\rho}+\sum_{A\in\mathcal{C}}|A|)$ space usage and $O(n^{\rho}\log n+T(n^{\rho},\log n,|Q|))$ expected query time, where $T(L,K,z)$ is the time it takes to create $L$ sketches of size $K$ for a set of size $z$. By providing a more efficient way to compute the sketches $S_{i}(A)$ we thus obtain a faster query time.

In order to create the sketches $S_{0}(A),\ldots,S_{L-1}(A)$ described above we first create a $L\times K$ table $T$ such that for each $i\in[L]$ and $j\in[K]$ we have $T[i,j]$ is a uniformly random integer chosen from $\{j\cdot t/K,\ldots,(j+1)\cdot t/K-1\}$, where $t$ is a parameter divisible by $K$ to be chosen later. The rows of the matrix are chosen independently. Each row is filled using a $2$-independent source of randomness. Now for a given $A\in[u]$ (or $Q$) we do as follows:

Let $S(A)$ be a size $t$ similarity sketch of Section 2. 2. 2.

For each $i\in[L]$ and $j\in[K]$ let $S_{i}(A)[j]$ be $S(A)[T[i,j]]$.

It follows that the time needed to create $S_{0}(A),\ldots,S_{L-1}(A)$ for any $A\in[u]$ is $O(LK+t\log t+|A|)$. We let $t=K\cdot\left\lceil{1+K\cdot\left(\frac{1}{j_{1}}-1\right)}\right\rceil$.

We start by bounding the number of “false positives”.

Lemma 6**.**

Let $A\in\mathcal{C}$ be such that $J(A,Q)\leq j_{2}$. Then for any $i\in[L]$ the probability that $S_{i}(A)=S_{i}(Q)$ is at most $\frac{1}{n}$.

Proof.

Fix $T[i,j]$ and let $v_{j}=T[i,j]$ for all $j\in[K]$. Now define $(X_{j})_{j\in[t]}$ as in Lemma 2. Then $S_{i}(A)=S_{i}(Q)$ if and only if $X_{v_{j}}=1$ for all $j\in[K]$, i.e. if $\prod_{j\in[K]}X_{v_{j}}=1$. By Lemma 2 this happens with probability at most $(J(A,Q))^{K}\leq j_{2}^{K}\leq 1/n$. ∎

Lemma 6 shows that for each $i\in[L]$ the expected number of sets $A\in\mathcal{C}$ with $J(A,Q)\leq j_{2}$ and $S_{i}(A)=S_{i}(Q)$ is at most $\left|\mathcal{C}\right|\cdot\frac{1}{n}=1$. Thus, the expected number of pairs $(i,A)\in[L]\times\mathcal{C}$ with $J(A,Q)\leq j_{2}$ and $S_{i}(A)=S_{i}(Q)$ is at most $L$.

Let $A_{0}\in\mathcal{C}$ be a set such that $J=J(A_{0},Q)\geq j_{1}$. We will give a lower bound on the probability that there exists an index $i\in[L]$ such that $S_{i}(A_{0})=S_{i}(Q)$. For $i\in[L]$ let $Y_{i}=[S_{i}(A_{0})=S_{i}(Q)]$ and let $Y=\sum_{i\in[L]}Y_{i}$. Using Lemma 2 and the same reasoning as in Lemma 6 we see that $\text{E}\!\left[{Y_{i}}\right]\geq\frac{(Jt)^{\underline{K}}}{t^{\underline{K}}}$. Using this we get:

[TABLE]

By the definition of $t$ we have that $\frac{K(1-J)}{J(t-K)}\leq\frac{1}{K}$. Hence $\text{E}\!\left[{Y_{i}}\right]\geq J^{K}\cdot\left(1-\frac{1}{K}\right)^{K}\geq J^{K}/4$. As a consequence we get that $\text{E}\!\left[{Y}\right]\geq L\cdot J^{K}/4\geq L\cdot j_{1}^{K}/4\geq\frac{1}{4}$, i.e. that the expected number of indices $i\in[L]$ such that $S_{i}(A_{0})=S_{i}(Q)$ is $\Omega(1)$. However, this does not suffice that such an index exists with constant probability. In order to prove this we will bound $\text{E}\!\left[{Y^{2}}\right]$ and use the inequality $\Pr\!\left[{Y>0}\right]\geq\frac{(\text{E}\!\left[{Y}\right])^{2}}{\text{E}\!\left[{Y^{2}}\right]}$, which follows from Cauchy-Schwarz’s Inequality.

Lemma 7**.**

Let $i_{0},i_{1}\in[L]$ be different indices. Then

[TABLE]

Proof.

The values $(T[i,j])_{(i,j)\in\left\{i_{0},i_{1}\right\}\times[L]}$ are all independent by definition. Let $R$ be the set containing these value, i.e.

[TABLE]

Define $X_{j}=[S(A_{0})[j]=S(Q)[j]]$ as in Lemma 2 and fix the value of $R$. Then by Lemma 2 $\text{E}\!\left[{Y_{i_{0}}Y_{i_{1}}\mid R}\right]\leq J^{\left|R\right|}$. It remains to understand $\left|R\right|$. For $j\in[K]$ let $Z_{j}=[T[i_{0},j]\neq T[i_{1},j]]$. Then it is easy to see that $\left|R\right|=K+\sum_{j\in[K]}Z_{j}$, that $(Z_{j})_{j\in[K]}$ are independent and that $\Pr\!\left[{Z_{j}=1}\right]=1-\frac{K}{t}$. Hence we can upper bound $\text{E}\!\left[{Y_{i_{0}}Y_{i_{1}}}\right]$ by

[TABLE]

Now

[TABLE]

and therefore $\text{E}\!\left[{Y_{i_{0}}Y_{i_{1}}}\right]\leq J^{2K}\cdot\left(1+\frac{K(1-J)}{Jt}\right)^{K}$. ∎

By the definition of $t$ we have $1+\frac{K(1-J)}{Jt}\leq 1+\frac{1}{K}\leq e^{1/K}$, and therefore Lemma 7 gives that $\text{E}\!\left[{Y_{i_{0}}Y_{i_{1}}}\right]\leq eJ^{2K}$. Hence:

[TABLE]

Since $\text{E}\!\left[{Y}\right]\leq L\cdot J^{K}$ we get that $\text{E}\!\left[{Y^{2}}\right]\leq L\cdot J^{K}+L^{2}\cdot e\cdot J^{2K}$. So the probability that $Y>0$ can be bounded below as follows:

[TABLE]

Avoiding false positives

We let $M=\left\{(i,A)\in[L]\times\mathcal{C}\mid S_{i}(A)=S_{i}(Q)\right\}$ be the set of matches. We have proved that for each $A_{0}\in\mathcal{C}$ with $J(A_{0},Q)\geq j_{1}$ with probability $\Omega(1)$ there exists $i\in[L]$ such that $(i,A_{0})\in M$. Furthermore, we have proved that the expected number of pairs $(i,A)\in M$ with $J(A,Q)\leq j_{2}$ is at most $L$. Naively, we could go through all the elements in $M$ until we find $(i,A)\in M$ such that $J(A,Q)>j_{2}$ in $O(\left|Q\right|)$ time per pair. The expected time would be $O\left(L\cdot\left|Q\right|\right)$, since in expectation we would check $\leq L$ pairs $(i,A)$ with $J(Q,A)\leq j_{2}$.

In order to obtain a expected running time of $O\left(L\cdot\left|Q\right|\right)$ we do something different. We split it into two cases depending on whether $\left|M\right|\geq CL$ or $\left|M\right|\leq CL$ for some sufficiently large constant $C$ depending on $j_{1},j_{2}$. We can in $O(L)$ time check if $\left|M\right|\geq CL$. First assume that $\left|M\right|\geq CL$. Then we find a subset $M^{\prime}\subseteq M$ of size $\left|M^{\prime}\right|=\left\lceil{CL}\right\rceil$, which we can clearly do in $O(L)$ time. Then we sample a uniformly random pair $(i,A)\in M^{\prime}$ and check if $J(A,Q)>j_{2}$. By Markov’s inequality the number of pairs $(i,A)\in M$ with $J(A,Q)\leq j_{2}$ is at most $\frac{CL}{2}$ with probability $\geq 1-\frac{2}{C}$, and in this case we find a set $A$ with $J(A,Q)\geq j_{2}$ with probability at least $\frac{1}{2}$. The time used in this case is clearly $O(L+\left|Q\right|)$.

Now assume that $\left|M\right|\leq CL$. We assume that we have made a similarity sketch of size $\Theta\left(\log n\right)$ for each set $A\in C$ and $Q$ - the running time and space usage for this is clearly dominated by what is used for the sketch of size $t$. For each $(i,A)\in M$ we now use Algorithm 2 with $\gamma=\frac{j_{1}+j_{2}}{2}$ on this sketch to separate $J(A,Q)$. We choose $r$ to be a sufficiently large constant. If the algorithm returns true we calculate $J(A,Q)$ and if it returns false we discard $A$. We note that for any set $A$ with $J(A,Q)\leq j_{2}$ the probability that we the algorithm returns true is at most $\frac{1}{n}$. Hence the expected number of sets $A\in\mathcal{C}$ with $J(A,Q)\leq j_{2}$ for which we calculate $J(A,Q)$ explicitly is at most $O(1)$. We conclude that the running time is $O(L+\left|Q\right|)$, since if we calculate $J(A,Q)$ for a set $A$ with $J(A,Q)>j_{2}$ we can terminate the algorithm and return $A$. Furthermore, if there exists a set $A_{0}\in\mathcal{C}$ with $J(A_{0},Q)\geq j_{1}$ the probability that the algorithm returns true is $\Omega(1)$ since $r$ is sufficiently large and so there is probability $\Omega(1)$ of finding a set with Jaccard similarity $>j_{2}$ in this case.

If there exists a set $A_{0}$ with Jaccard similarity $J(A_{0},Q)\geq j_{1}$ we conclude that the probability of finding a set $A$ with $J(A,Q)>j_{2}$ is therefore at least $\Omega(1)-\frac{2}{C}$. By choosing $C$ sufficiently large we ensure that $\Omega(1)-\frac{2}{C}=\Omega(1)$.

Summarizing we get the following theorem.

Theorem 2**.**

Let $0<j_{2}<j_{1}<1$ be constants, and let $\rho=\frac{\log(1/j_{1})}{\log(1/j_{2})}$. Let $U$ be a set of elements and let $\mathcal{C}$ be collection of $n$ sets from $U$. Then there exists a data structure using space $O\left(n^{1+\rho}+\sum_{A\in\mathcal{C}}\left|A\right|\right)$ and has query time $O\left(n^{\rho}\log n+\left|Q\right|\right)$ such that: Given a set $Q$ if there exists a set $A_{0}\in\mathcal{C}$ with $J(A_{0},Q)\geq j_{1}$, then with constant probability the data structure returns a set $A\in\mathcal{C}$ with $J(A_{0},Q)>j_{2}$.

Appendix A Proof of Lemma 5

Proof.

For $i=0,1,2,\ldots,n$ we let $X_{<i}=\sum_{j<i}X_{j}$.

First we prove the bound on the expected value of $\tau_{F}$. For $j=r,r+1,\ldots,t$ let $T_{j}=j$ if the algorithm returns false when $i=j$ in the loop and let $T_{j}=0$ otherwise. We clearly have that $\tau_{F}=r+\sum_{i=r}^{t}T_{i}$. Clearly $\sum_{i=r}^{2r}T_{i}\leq 2r$ by definition, and therefore

[TABLE]

Now fix $i>2r$. If $T_{i}=i$ then we must have that $X_{<i}\geq i\cdot\gamma+\sqrt[3]{i^{2}}$. Let $j=\left\lfloor i/2\right\rfloor$. Since the algorithm did not stop earlier we must also have $X_{<j}<j\cdot\gamma+\sqrt[3]{j^{2}}$. Hence we have that:

[TABLE]

Let $\gamma^{\prime}=\gamma+\frac{1}{2}\cdot\left(\frac{1}{\sqrt[3]{i}}+\frac{1}{\sqrt[3]{j}}\right)$. If $p\leq\gamma^{\prime}$ then we see that if $X_{<i}\geq i\cdot\gamma+\sqrt[3]{i^{2}}$ then:

[TABLE]

And by Hoeffding’s inequality we conclude that:

[TABLE]

If $p>\gamma^{\prime}$ we conclude in the same manner that $\Pr\!\left[{X_{<j}<j\cdot\gamma+\sqrt[3]{j^{2}}}\right]\leq e^{-\Omega(\sqrt[3]{i})}$. Hence we get that:

[TABLE]

We conclude that $\text{E}\!\left[{\tau_{F}}\right]=O(r)$ as desired.

We now assume that $p\geq\gamma+\delta$ and prove that (2) is a lower bound on the probability that true is returned. By a union bound and Hoeffding’s inequality we get that the probability that false is returned is at most

[TABLE]

as desired.

Now assume that $p\geq\gamma-\delta$. We note that true is only returned if $X_{<t}>t\cdot\gamma$. By Hoeffding’s inequality this happens with probability at most

[TABLE]

showing that (3) holds. ∎