Finding Associations and Computing Similarity via Biased Pair Sampling

TL;DR

This paper introduces a biased sampling method for efficiently finding associations and computing similarities in data, especially effective with many items per transaction, outperforming exact methods in speed and resource usage.

Contribution

The paper presents a simple biased sampling technique supporting various similarity measures and high-confidence rules, with theoretical and empirical advantages over exact methods.

Findings

Significant speedups in association mining tasks.

Over an order of magnitude reduction in computation time.

Effective handling of transactions with many items.

Abstract

This version is ***superseded*** by a full version that can be found at http://www.itu.dk/people/pagh/papers/mining-jour.pdf, which contains stronger theoretical results and fixes a mistake in the reporting of experiments. Abstract: Sampling-based methods have previously been proposed for the problem of finding interesting associations in data, even for low-support items. While these methods do not guarantee precise results, they can be vastly more efficient than approaches that rely on exact counting. However, for many similarity measures no such methods have been known. In this paper we show how a wide variety of measures can be supported by a simple biased sampling method. The method also extends to find high-confidence association rules. We demonstrate theoretically that our method is superior to exact methods when the threshold for "interesting similarity/confidence" is above the average pairwise similarity/confidence, and the average support is not too low. Our method is particularly good when transactions contain many items. We confirm in experiments on standard association mining benchmarks that this gives a significant speedup on real data sets (sometimes much larger than the theoretical guarantees). Reductions in computation time of over an order of magnitude, and significant savings in space, are observed.

Figures

Equations

$$|S_i\cap S_j|f(|S_i|,|S_j|,s(i,j))=1.$$

$$|S_i\cap S_j|f(|S_i|,|S_j|,\Delta )\mu .$$

$$\frac{|S_i\cap S_j|}{|S_i]+|S_j|}\frac{1+\Delta }{\Delta }\mu =\mu \frac{s(i,j)(1+\Delta )}{(1+s(i,j))\Delta }\le 2\mu s(i,j)/\Delta ,$$

$$|S_i\cap S_j|f(|S_i|,|S_j|,\Delta )=O(s(i,j)/\Delta ).$$

$$O\left(mb\log n+\frac{\mu }{\Delta }\sum _{1\le i<j\le n}s(i,j)\right).$$

$$\sum _{i=1}^n\sum _{j=i+1}^n{p}_i+p_j\le \sum _{i=1}^n{p}_{i}n+\sum _{j=1}^{n}n{p}_j=2n.$$

$$M(i,j)>\mu /2\text{ or }M(i,j)f(c(i),c(j),\Delta )\ge 1,$$

$$O\left(\frac{N+N^{\prime }}{B}{\log }_{\frac{M}{B}}\left(\frac{N+N^{\prime }}{M}\right)\right)\text{ I/Os }.$$

Full text

I Introduction

A central task in data mining is finding associations in a binary relation. Typically, this is phrased in a “market basket” setup, where there is a sequence of baskets (from now on “transactions”), each of which is a set of items. The goal is to find patterns such as “customers who buy diapers are more likely to also buy beer”. There is no canonical way of defining whether an association is interesting — indeed, this seems to depend on problem-specific factors not captured by the abstract formulation. As a result, a number of measures exist: In this paper we deal with some of the most common measures, including Jaccard [1], lift [2, 3], cosine, and all_confidence [4, 5]. In addition, we are interested in high-confidence association rules, which are closely related to the overlap coefficient similarity measure. We refer to [6, Chapter 5] for general background and discussion of similarity measures.

In the discussion we limit ourselves to the problem of binary associations, i.e., patterns involving pairs of items. There is a large literature considering the challenges of finding patterns involving larger item sets, taking into account the aspect of time, multiple-level rules, etc. While some of our results can be extended to cover larger item sets, we will for simplicity concentrate on the binary case. Previous methods rely on one of the following approaches:

Identifying item pairs $(i,j)$ that “occur frequently together” in the transactions — in particular, this means counting the number of co-occurrences of each such pair — or 2. 2.

Computing a “signature” for each item such that the similarity of every pair of items can be estimated by (partially) comparing the item signatures.

Our approach is different from both these approaches, and generally offers improved performance and/or flexibility. In some sense we go directly to the desired result, which is the set of pairs of items with similarity measure above some user-defined threshold $\Delta$. Our method is sampling based, which means that the output may contain false positives, and there may be false negatives. However, these errors are rigorously understood, and can be reduced to any desired level, at some cost of efficiency — our experimental results are for a false negative probability of less than 2%. The method for doing sampling is the main novelty of this paper, and is radically different from previous approaches that involve sampling.

The main focus in many previous association mining papers has been on space usage and the number of passes over the data set, since these have been recognized as main bottlenecks. We believe that time has come to also carefully consider CPU time. A transaction with $b$ items contains $\binom{b}{2}$ item pairs, and if $b$ is not small the effort of considering all pairs is non-negligible compared to the cost of reading the item set. This is true in particular if data resides in RAM, or on a modern SSD that is able to deliver data at a rate of more than a gigabyte per second. One remedy that has been used (to reduce space, but also time) is to require high support, i.e., define “occur frequently together” such that most items can be thrown away initially, simply because they do not occur frequently enough (they are below the support threshold). However, as observed in [1] this means that potentially interesting or useful associations (e.g. correlations between genes and rare diseases) are not reported. In this paper we consider the problem of finding associations without support pruning. Of course, support pruning can still be used to reduce the size of the data set before our algorithms are applied.

In the following sections we first discuss the need for focusing on CPU time in data mining, and then elaborate on the relationship between our contribution and related works.

I-A I/O versus CPU

In recent years, the capacity of very fast storage devices has exploded. A typical desktop computer has 4–16 GB of RAM, that can be read (sequentially) at a speed of at least 800 million 32-bit words per second. The flash-based ioDrive Duo of Fusion-io offers up to over a terabyte of storage that can be read at around 400 million 32-bit words per second. Thus, even massive data sets can be read at speeds that make it challenging for CPUs to keep up. An 8-core system must, for example, process 100 million (or 50 million) items per core per second. At 3 GHz this is 33 clock cycles (or 66 clock cycles) per item. This means that any kind of processing that is not constant time per item (e.g., using time proportional to the size of the transaction containing the item) is likely to be CPU bound rather than I/O bound. For example, a hash table lookup requires on the order of 5-10 ns even if the hash table is L2 cache-resident (today less than 10 MB per core). This gives an upper limit of 100-200 million lookups per second in each core, meaning that any algorithm that does more than a dozen hash table operations per item (e.g. updating the count of some item pairs) is definitely CPU bound, rather than I/O bound. In conclusion, we believe it is time to carefully consider optimizing internal computation time, rather than considering all computation as “free” by only counting I/Os or number of passes. Once CPU efficient algorithms are known, it is likely that the remaining bottleneck is I/O. Thus, we also consider I/O efficient versions of our algorithm.

I-B Previous work

Exact counting of frequent item sets

The approach pioneered by the A-Priori algorithm [7, 8], and refined by many others (see e.g. [9, 10, 11, 12, 13]), allows, as a special case, finding all item pairs $(i,j)$ that occur in more than $k$ transactions, for a specified threshold $k$. However, for the similarity measures we consider, the value of $k$ must in general be chosen as a low constant, since even pairs of very infrequent items can have high similarity. This means that such methods degenerate to simply counting the number of occurrences of all pairs, spending time $\Theta(b^{2})$ on a transaction with $b$ items. Also, generally the space usage of such methods (at least those requiring a constant number of passes over the data) is at least 1 bit of space for each pair that occurs in some transaction.

The problem of counting the number of co-occurrences of all item pairs is in fact equivalent to the problem of multiplying sparse 0-1 matrices. To see this, consider the $n\times m$ matrix A in which each row Ai is the incidence vector having $1$ in position $p$ iff the $i$th element in the set of items appears in the $p$th transaction. Each entry $\tilde{\textsc{A}}_{i,j}$ of the $n\times n$ matrix $\tilde{\textsc{A}}=\textsc{A}\times\textsc{A}^{T}$ represents the number of transactions in which the pair $(i,j)$ appears. The best theoretical algorithms for (sparse) matrix multiplication [14, 15, 16] scale better than the A-Priori family of methods as the transaction size gets larger, but because of huge constant factors this is so far only of theoretical interest.

Sampling transactions

Toivonen [17] investigated the use of sampling to find candidate frequent pairs $(i,j)$: Take a small, random subset of the transactions and see what pairs are frequent in the subset. This can considerably reduce the memory used to actually count the number of occurrences (in the full set), at the cost of some probability of missing a frequent pair. This approach is good for high-support items, but low-support associations are likely to be missed, since few transactions contain the relevant items.

Locality-sensitive hashing

Cohen et al. [1] proposed the use of another sampling technique, called min-wise independent hashing, where a small number of occurrences of each item (a “signature”) is sampled. This means that occurrences of items with low support are more likely to be sampled. As a result, pairs of (possibly low-support) items with high jaccard coefficient are found — with a probability of false positives and negatives. A main result of [1] is that the time complexity of their algorithm is proportional to the sum of all pairwise jaccard coefficients, plus the cost of initially reading the data. Our main result has basically the same form, but has the advantage of supporting a wide class of similarity measures.

Min-wise independent hashing belongs to the class of locality-sensitive hashing methods [18]. Another such method was described by Charikar [19], who showed how to compute succinct signatures whose Hamming distance reflects angles between incidence vectors. This leads to an algorithm for finding item pairs with cosine similarity above a given threshold (again, with a probability of false positives and negatives), that uses linear time to compute the signatures, and $\Theta(n^{2})$ time to find the similar pairs, where $n$ is the number of distinct items in all transactions. Charikar also shows that many similarity measures, including some measures supported by our algorithm, cannot be handled using the approach of locality-sensitive hashing.

Deterministic signature methods

In the database community, finding all pairs with similarity above a given threshold is sometimes referred to as a “similarity join.” Recent results on similarity joins include [20, 21, 22, 23]. While not always described in this way, these methods can be seen as deterministic analogues of the locality-sensitive hashing methods, offering exact results. The idea is to avoid computing the similarity of every pair by employing succinct “signatures” that may serve as witnesses for low similarity. Most of these methods require the signatures of every pair of items to be (partially) compared, which takes $\Omega(n^{2})$ time. However, the worst-case asymptotic performance appears to be no better than the A-Priori family of methods. A similarity join algorithm that runs faster than $\Omega(n^{2})$ in some cases is described in [20]. However, this algorithm exhibits a polynomial dependence on the maximum number $k$ of differences between two incidence vectors that are considered similar, and for many similarity measures the relevant value of $k$ may be linear in the number $m$ of transactions.

I-C Our results

In this paper we present a novel sampling technique to handle a variety of measures (including jaccard, lift, cosine, and all_confidence), even finding similar pairs among low support items. The idea is to sample a subset of all pairs $(i,j)$ occurring in the transactions, letting the sampling probability be a function of the supports of $i$ and $j$. For a parameter $\mu$, the probability is chosen such that each pair with similarity above a threshold $\Delta$ (an “interesting pair”) will be sampled at least $\mu$ times, in expectation, while we do not expect to see a pair $(i,j)$ whose measure is significantly below $\Delta$. A naïve implementation of this idea would still use quadratic time for each transaction, but we show how to do the sampling in near-linear time (in the size of the transaction and number of sampled pairs).

The number of times a pair is sampled follows a binomial distribution, which allows us to use the sample to infer which pairs are likely to have similarity above the threshold, with rigorous bounds on false negative and false positive probabilities. We show that the time used by our algorithm is (nearly) linear in the input size and in the the sum of all pairwise similarities between items, divided by the threshold $\Delta$. This is (close to) the best complexity one could hope for with no conditions on the distribution of pairwise similarities. Under reasonable assumptions, e.g. that the average support is not too low, this gives a speedup of a factor $\Omega(b/\log b)$, where $b$ is the average size of a transaction.

We show in extensive experiments on standard data sets for testing data mining algorithms that our approach (with a $1.8\%$ false negative probability) gives speedup factors in the vicinity of an order of magnitude, as well as significant savings in the amount of space required, compared to exact counting methods. We also present evidence that for data sets with many distinct items, our algorithm may perform significantly less work than methods based on locality-sensitive hashing.

I-D Notation

Let $T_{1},\dots,T_{m}$ be a sequence of transactions, $T_{j}\subseteq[n]$. For $i=1,\dots,n$ let $S_{i}=\{j\;|\;i\in T_{j}\}$, i.e., $S_{i}$ is the set of occurrences of item $i$.

We are interested in finding associations among items, and consider a framework that captures the most common measures from the data mining literature. Specifically, we can handle a similarity measure $s(i,j)$ if there exists a function $f:{\bf N}\times{\bf N}\times{\bf R}_{+}\rightarrow{\bf R}_{+}$ that is non-increasing in all parameters, and such that:

[TABLE]

In other words, the similarity should be the solution to an equation of the form given above. Fig. 1 shows particular measures that are special cases. The monotonicity requirements on $f$ hold for any reasonable similarity measure: increasing $|S_{i}\cap S_{j}|$ should not decrease the similarity, and adding an occurrence of $i$ or $j$ should not increase the similarity unless $|S_{i}\cap S_{j}|$ increases. In the following we assume that $f$ is computable in constant time, which is clearly a reasonable assumption for the measures of Fig. 1.

II Our algorithm

The goal is to identify pairs $(i,j)$ where $s(i,j)$ is “large”. Given a user-defined threshold $\Delta$ we want to report the pairs where $s(i,j)\geq\Delta$. We observe that all measures in Fig. 1 are symmetric, so it suffices to find all pairs $(i,j)$ where $|S_{i}|\leq|S_{j}|$, $i\neq j$, and $s(i,j)\geq\Delta$.

II-A Algorithm idea

Our algorithm is randomized and finds each qualifying pair with probability $1-\varepsilon$, where $\varepsilon>0$ is a user-defined error probability. The algorithm may also return some false positives, but each false positive pair is likely to have similarity within a small constant factor of $\Delta$. If desired, the false positives can be reduced or eliminated in a second phase, but we do not consider this step here.

The basic idea is to randomly sample pairs of items that occur together in some transaction such that for any pair $(i,j)$ the expected number times it is sampled is a strictly increasing function of $s(i,j)$. Indeed, in all cases except the jaccard measure it is simply proportional to $s(i,j)$. We scale the sampling probability such that for all pairs with $s(i,j)\geq\Delta$ we expect to see at least $\mu$ occurrences of $(i,j)$, where $\mu$ is a parameter (defined later) that determines the error probability.

II-B Implementation

Fig. 2 shows our algorithm, called BiSam (for biased sampling). The algorithm iterates through the transactions, and for each transaction $T_{t}$ adds a subset of $T_{t}\times T_{t}$ to a multiset $M$ in time that is linear in $|T_{t}|$ and the number of pairs added. We use $T_{t}[i]$ to denote the $i$th item in $T_{t}$. Because $f$ is non-increasing and $T_{t}$ is sorted according to the order induced by $c(\cdot)$ we will add $(T_{t}[i],T_{t}[j])\in T_{t}\times T_{t}$ if and only if $f(c(T_{t}[i]),c(T_{t}[j]),\Delta)\mu>r$. The second loop of the algorithm builds an output set containing those pairs $(i,j)$ that either occur at least $\mu/2$ times in $M$, or where the number of occurrences in $M$ imply that $s(i,j)\geq\Delta$ (with probability $1$).

The best implementation of the subprocedure ItemCount depends on the relationship between available memory and the number $n$ of distinct items. If there is sufficient internal memory, it can be efficiently implemented using a hash table. For larger instances, a sort-and-count approach can be used (Section III-B). The multiset $M$ can be represented using a hash table with counters (if it fits in internal memory), or more generally by an external memory data structure. In the following we first consider the standard model (often referred to as the “RAM model”), where the hash tables fit in internal memory, and assume that each insertion takes constant time. Then we consider the I/O model, for which an I/O efficient implementation is discussed. As we will show in Section IV, a sufficiently large value of $\mu$ is $8\ln(1/\varepsilon)$. Fig. 5 shows more exact, concrete values of $\mu$ and corresponding false positive probabilities $\varepsilon$.

Example. Suppose ItemCount has been run and the supports of items 1–6 are as shown in Fig. 3.

Suppose now that the transaction $T_{t}=\{6,5,4,3,2,1\}$ is given. Note that its items are written according to the number of occurrences of each item. Assuming the similarity measure is cosine, $\mu=10$, $\Delta=0.7$, and $r$ for this transaction equal to 0.9, our algorithm would select from $T_{t}\times T_{t}$ the pairs shown in Fig. 4.

Suppose that after processing all transactions the pair $(6,5)$ occurs 3 times in $M$, $(6,4)$ occurs twice in $M$, $(6,1)$ occurs once in $M$, and $(5,4)$ occurs 4 times in $M$. Then the algorithm would output the pairs: $(6,5)$ (since $M(6,5)<\mu/2$ but $M(6,5)f(3,5,0.7)>1$), and $(5,4)$ (same situation as before).

III Analysis of running time

Our main lemma is the following:

Lemma 1

For all pairs $(i,j)$, where $i\neq j$ and $c(i)\leq c(j)$, if $f(c(i),c(j),\Delta)\mu<1$ then at the end of the procedure, $M(i,j)$ has binomial distribution with $|S_{i}\cap S_{j}|$ trials and mean

[TABLE]

If $f(c(i),c(j),\Delta)\mu\geq 1$ then at the end of the procedure $M(i,j)=|S_{i}\cap S_{j}|$.

Proof:

As observed above, the algorithm adds the pair $(i,j)$ to $M$ in iteration $t$ if and only if $(i,j)\in T_{t}\times T_{t}$ and $f(c(i),c(j),\Delta)\mu>r$, where $r$ is the random number in $[0;1)$ chosen in iteration $t$. This means that for every $t\in S_{i}\cap S_{j}$ we add $(i,j)$ to $M$ with probability $\min(1,f(c(i),c(j),\Delta)\mu)$. In particular, $M(i,j)=|S_{i}\cap S_{j}|$ for $f(c(i),c(j),\Delta)\mu\geq 1$. Otherwise, since the value of $r$ is independently chosen for each $t$, the distribution of $M(i,j)$ is binomial with $|S_{i}\cap S_{j}|$ trials and mean $|S_{i}\cap S_{j}|f(c(i),c(j),\Delta)\mu$. ∎

Looking at Fig. 1 we notice that for the jaccard similarity measure $s(i,j)=\frac{|S_{i}\cap S_{j}|}{|S_{i}]+|S_{j}|-|S_{i}\cap S_{j}|}$, the mean of the distribution is

[TABLE]

where the inequality uses $s(i,j),\Delta\in[0;1]$. For all other similarity measures the mean of the binomial distribution is $\mu\,s(i,j)/\Delta$. As a consequence, for all these measures, pairs with similarity below $(1-\varepsilon)\Delta$ will be counted exactly, or sampled with mean $(1-\Omega(\varepsilon))\mu$. Also notice that for all the measures we consider,

[TABLE]

We provide a running time analysis both in the standard (RAM) model and in the I/O model of Aggarwal and Vitter [24]. In the latter case we present an external memory efficient implementation of the algorithm, IOBiSam. Let $b$ denote the average number of items in a transaction, i.e., there are $bm$ items in total. Also, let $z$ denote the number of pairs reported by the algorithm.

III-A Running time in the standard model

The first and last part of the algorithm clearly runs in expected time $O(mb+z)$. The time for reporting the result is dominated by the time used for the main loop, but analyzing the complexity of the main loop requires some thought. The sorting of a transaction with $b_{1}$ items takes $O(b_{1}\log b_{1})$ time, and in particular the total cost of all sorting steps is $O(mb\log n)$.111We remark that if $O(mb)$ internal memory is available, two applications of radix sorting could be used to show a theoretically stronger result, by sorting all transactions in $O(mb)$ time, following the same approach as the external memory variant.

What remains is to account for the time spent in the while loop. We assume that $|S_{i}\cap S_{j}|f(|S_{i}|,|S_{j}|,\Delta)=O(s(i,j)/\Delta)$, which is true for all the measures we consider. The time spent in the while loop is proportional to the number of items sampled, and according to Lemma 1 the pair $(i,j)$ will be sampled $|S_{i}\cap S_{j}|f(|S_{i}|,|S_{j}|,\Delta)\mu=O(\mu\,s(i,j)/\Delta)$ times in expectation if $f(c(i),c(j),\Delta)\mu<1$, and $|S_{i}\cap S_{j}|$ times otherwise. In both cases, the expected number of samples is $O(s(i,j)\frac{\mu}{\Delta})$. Summing over all pairs we get the total time complexity.

Theorem 2

Suppose we are given transactions $T_{1},\dots,T_{m}$, each a subset of $[n]$, with $mb$ items in total, and that $f$ is the function corresponding to the similarity measure $s$. Also assume that

[TABLE]

Then the expected time complexity of BiSam$(T_{1},\dots,T_{m};f,\mu,\Delta)$ in the standard model is:

[TABLE]

Discussion

This result is close to the best we could hope for with no condition on the distribution of pairwise similarities. The first term is near-linear in the input size, and the output size $z$ may be as large as $\Omega(\Delta^{-1}\sum_{1\leq i<j\leq n}s(i,j))$. This happens if the average similarity among the pairs reported is $O(\Delta)$, and the total similarity among other pairs is low and does not dominate the sum. For such inputs, the algorithm runs in $O(mb\log n+\mu z)$ time, and clearly $\Omega(mb+z)$ time is needed by any algorithm.

A comparison can be made with the complexity of schemes counting the occurrences of all pairs. Such methods use time $\Omega(mb^{2})$, which is a factor $b/\log n$ larger than the first term. In fact, the difference will be larger if the distribution of transaction sizes is not even — and in particular the difference in time will be at least a factor $b/\log b$ (but this requires a more thorough analysis). Since ususally one is interested in reporting the highly similar pairs, the condition that $\Delta$ is greater than the average similarity $\sum_{1\leq i<j\leq n}s(i,j)/\binom{n}{2}$ is frequently true. (In fact, one could imagine that $\Delta$ would in many cases be much greater than the average similarity.) From the above we can obtain the following simple (in some cases pessimistic) upper bound on the time complexity:

Corollary 3

If $\Delta$ is not chosen smaller than the average pairwise similarity, the expected time complexity of BiSam is $O(mb\log n+\mu n^{2})$.

This means that under the assumption of the corollary we win a factor of at least $\min(b/\log b,\tfrac{m}{\mu}(\tfrac{b}{n})^{2})$ compared to the exact counting approach. Note in this context that $\mu$ can be chosen as a small value (e.g., $\mu=15$ in our experiments). In most of our experiments the first of the two terms (the counting phase) dominated the time complexity. However, we also found that for some data sets with mainly low-support items, the second term dominated. If we let $\sigma=mb/n$ denote the average support, the speedup can be expressed as $\Omega(b\min(1/\log b,\tfrac{\sigma}{\mu n}))$. That is, the second term dominates if the average support is below roughly $\mu n/\log b$.

Independent items

As further evidence for (or explanation of) why the time complexity of the second term may be close to linear, we consider an input where each item $i$ appears in a given transaction with probability $p_{i}$, independently of all other items. Thus, the probability that distinct items $i$ and $j$ appear in a transaction is $p_{i}p_{j}$. We observe that each similarity measure $s(i,j)$ in Fig. 1, with the exception of lift, satisfies $s(i,j)\leq\bar{s}(i,j)$, where $\bar{s}(i,j)=\frac{|S_{i}\cap S_{j}|}{|S_{i}|}+\frac{|S_{i}\cap S_{j}|}{|S_{j}|}$. Thus, we get an upper bound on running time for these measures by considering the similarity measure $\bar{s}(i,j)$. Observe that the expected value of $\bar{s}(i,j)$ is $p_{i}+p_{j}$ by linearity of expectation. Hence, the expected sum of similarities is:

[TABLE]

This means that the running time of BiSam is indeed $O(mb\log n+n/\Delta)$ for independent items.

III-B Running time in the I/O model

We now present IOBiSam, an I/O efficient implementation of the BiSam algorithm. The rest of the paper can be read independently of this section. As before, we assume that the similarity measure is such that $|S_{i}\cap S_{j}|f(|S_{i}|,|S_{j}|,\Delta)=O(s(i,j)/\Delta)$

In order to compute the support of each item, which means computing the ItemCount function, a sorting of the dataset’s items is carried out. It is necessary to keep track of which transaction each item belongs to. To compute the sorted list of items, O$(\frac{N}{B}\log_{\frac{M}{B}}\frac{N}{M})$ I/Os are needed [24], where $N=mb$ is the number of pairs $c=\langle$item, Transaction ID$\rangle$, $M$ is the number of such pairs that fit in memory, and $B$ is the number of pairs that fit in a memory page. When the items are sorted, it is trivial to compute the number of occurrences of each item, so it takes just $O(\frac{N}{B})$ I/Os to compute and store the tuples $c\langle$item,support,Transaction ID$\rangle$. In the following, let $\tilde{C}$ be the set of such tuples written to disk.

We then sort the tuples according to transaction ID, and secondarily according to support, again using $O(\frac{N}{B}\log_{\frac{M}{B}}\frac{N}{M})$ I/Os. This gives us each transaction in sorted order, according to item supports. Assuming that each transaction fits in main memory222The assumption is made only for simplicity of exposition, since the result holds also without this assumption. it is simple to determine which pairs satisfy the inequality $f(c(T_{t}[i]),c(T_{t}[j]),\Delta)\mu>r$. When a pair satisfies the inequality, it is buffered in an output page in memory, together with the item supports. Once the page is filled, it is flushed to external memory. The total cost of this phase is $O(\frac{N+N^{\prime}}{B})$ I/Os for the flushings and reads, where $N^{{}^{\prime}}$ is the total number of pairs satisfying the inequality (i.e., the number of samples taken). As before, the expected value of $N^{\prime}$ is $O(\frac{\mu}{\Delta}\sum_{1\leq i<j\leq n}s(i,j))$. Finally, we spend $O(\frac{N^{\prime}}{B}\log_{\frac{M}{B}}\frac{N^{\prime}}{M})$ I/Os to sort the sampled pairs (according to e.g. lexicographical order). Then it is easy to compute $M(i,j)$, i.e., the number of times each pair $(i,j)$ has been sampled by the algorithm, using $O(\frac{N^{{}^{\prime}}}{B})$ I/Os. The final step is to output all the pairs satisfying the condition:

[TABLE]

which needs $O(\frac{N^{\prime}}{B})$ I/Os. We observe that this cost is dominated by the cost of previous operations. The most expensive steps are the sorting steps, whose total input has size $O(N+N^{\prime})$, implying that the following theorem holds:

Theorem 4

Suppose we are given transactions $T_{1},\dots,T_{m}$, each a subset of $[n]$, with $N=mb$ items in total, and $f$ is the function corresponding to the similarity measure $s$. Also assume $|S_{i}\cap S_{j}|f(|S_{i}|,|S_{j}|,\Delta)=O(s(i,j)/\Delta)$. For $N^{\prime}=O(\frac{\mu}{\Delta}{\sum_{1\leq i<j\leq n}s(i,j)})$, the expected complexity of IOBiSam$(T_{1},\dots,T_{m};f,\mu,\Delta)$ in the I/O model is

[TABLE]

IV Analysis of error probability

False negatives. We first bound the probability that a pair $(i,j)$ with $s(i,j)\geq\Delta$ is not reported by the algorithm. This happens if $M(i,j)\leq\mu/2$ and $M(i,j)f(c(i),c(j),\Delta)<1$. If $f(c(i),c(j),\Delta)\mu\geq 1$ then the pair $(i,j)$ is reported with probability $1$. Otherwise, since $M(i,j)$ has binomial distribution, it follows from Chernoff bounds (see e.g. [25, Theorem 4.2] with $\delta=1/2$) that the probability of the former event is at most $\exp(-\delta^{2}\mu/2)=\exp(-\mu/8)$. Solving for $\mu$ this means that we have error probability at most $\varepsilon$ if $\mu\geq 8\ln(1/\varepsilon)$. This bound is pessimistic, especially when $\varepsilon$ is not very small. Tighter bounds can be obtained using the Poisson approximation to the binomial distribution, which is known to be precise when the number of trials is not too small (e.g., at least 100). Fig. 5 shows some values of $\mu$ and corresponding false negative probabilities, using the Poisson approximation.

False positives. The probability that a pair $(i,j)$ with $s(i,j)<\Delta$ is reported depends on how far the mean $|S_{i}\cap S_{j}|f(|S_{i}|,|S_{j}|,\Delta)\mu$ is from $\mu$. With the exception of the jaccard measure, all measures we consider have mean $\mu\,s(i,j)/\Delta$. In the following we assume this is the case (a slightly more involved analysis can be made for the jaccard measure). If the ratio $s(i,j)/\Delta$ is close to $1$, there is a high probability that the pair will be reported. However, this is not so bad since $s(i,j)$ is close to the threshold $\Delta$. On the other hand, when $s(i,j)/\Delta$ is close to zero we would like the probability that $(i,j)$ is reported to be small. Again, we may use the fact that either $f(c(i),c(j),\Delta)\mu\geq 1$ (in which case the pair is exactly counted and reported with probability [math]), or $M(i,j)$ has binomial distribution with mean $s(i,j)\frac{\mu}{\Delta}$. For $s(i,j)<\Delta/2$ we can use Chernoff bounds, or the Poisson approximation, to bound the probability that $M(i,j)>\mu/2$. Fig. 6 illustrates two Poisson distributions (one corresponding to an item pair with measure three times below the threshold, and one corresponding to an item pair with measure at the threshold).

Actually, the number $\mu/2$ in the reporting loop of the BiSam algorithm is just one possible choice in a range of possible trade-offs between the number of false positives and false negatives. As an alternative to increasing this threshold, a post-processing procedure may efficiently eliminate most false positives by more accurately estimating the corresponding values of the measure.

V Variants and extensions

In this section we mention a number of ways in which our results can be extended.

V-A Alternative filtering of false positives

The threshold of $\mu/2$ in the BiSam algorithms means that we filter away most pairs whose similarity is far from $\Delta$. An alternative is to spend more time on the pairs $(i,j)\in M$, using a sampling method to obtain a more accurate estimate of $|S_{i}\cap S_{j}|$. A suitable technique could be to use min-wise independent hash functions [26, 27] to obtain a sketch of each set $S_{i}$. It suffices to compare two sketches in order to have an approximation of the jaccard similarities of $S_{i}$ and $S_{j}$, which in turn gives an approximation of $|S_{i}\cap S_{j}|$. Based on this we may decide if a pair is likely to be interesting, or if it is possible to filter it out. The sketches could be built and maintained during the ItemCount procedure using, say, a logaritmic number of hash functions. Indyk [27] presents an efficient class of (almost) min-wise independent hash functions.

For some similarity measures such as lift and overlap coefficient the similarity of two sets may be high even if the sets have very different sizes. In such cases, it may be better to sample the smaller set, say, $S_{i}$, and use a hash table containing the larger set $S_{j}$ to estimate the fraction $|S_{i}\cap S_{j}|/|S_{i}|$.

V-B Reducing space usage by using counting Bloom filters

At the cost of an extra pass over the data, we may reduce the space usage. The idea, previously found in e.g. [12], is to initially create an approximate representation of $M$ using counting Bloom filters (see [28] for an introduction). Then, in a subsequent pass we may count only those pairs that, according to the approximation, may occur at least $\mu/2$ times.

V-C Weighted items

Some applications of the cosine measure, e.g. in information retrieval, require the items to be weighted. BiSam easily extends to this setting.

V-D Adaptive variant.

Instead of letting $\Delta$ be a user-defined variable, we may (informally) let $\Delta$ go from $\infty$ towards [math]. This can be achieved by maintaining a priority queue of item pairs, where the priority reflects the value of $\Delta$ that would allow the pair to be sampled. Because $f$ is non-increasing in all parameters it suffices to have a linear number of pairs from each transaction in the priority queue at any time, namely the pairs that are “next in line” to be sampled. For each of the similarity measures in Fig. 1 the value of $\Delta$ for a pair $(i,j)$ is easily computed by solving the equation $f(|S_{i}|,|S_{j}|,s)\mu=r$ for $s$. Decreasing $\Delta$ corresponds to removing the pair with the maximum value from the priority queue. At any time, the set of sampled item pairs will correspond exactly to the choice of $\Delta$ given by the last pair extracted from the priority queue. The procedure can be stopped once sufficiently many results have been found.

V-E Composite measures

Notice that if $f_{1}(|S_{i}|,|S_{j}|,\Delta)$ and $f_{2}(|S_{i}|,|S_{j}|,\Delta)$ are both non-increasing, then any linear combination $\alpha f_{1}+\beta f_{2}$, where $\alpha,\beta>0$, is also non-increasing. Similarly, $\min(\alpha f_{1},\beta f_{2})$ is non-increasing. This allows us to use BiSam to directly search for pairs with high similarity according to several measures (corresponding to $f_{1}$ and $f_{2}$), e.g., pairs with cosine similarity at least $0.7$ and lift at least $2$.

VI Experiments

To make experiments fully reproducible and independent of implementation details and machine architecture, we focus our attention on the number of hash table operations, and the number of items in the hash tables. That is, the time for BiSam is the number of items in the input set plus the number of pairs inserted in the multiset $M$. The space of BiSam is the number of distinct items (for support counts) plus the number of distinct pairs in $M$. Similarly, the time for methods based on exact counting is the number of items in the input set plus the number of pairs in all transactions (since every pair is counted), and the space for exact counting is the number of distinct items plus the number of distinct pairs that occur in some transaction.

We believe that these simplified measures of time and space are a good choice for two reasons. First, hash table lookups and updates require hundreds of clock cycles unless the relevant key is in cache. This means that a large fraction of the time spent by a well-tuned implementation is used for hash table lookups and updates. Second, we are comparing two approaches that have a similar behavior in that they count supports of items and pairs. The key difference thus lies in the number of hash table operations, and the space used for hash tables. Also, this means that essentially any speedup or space reduction applicable to one approach is applicable to the other (e.g. using counting Bloom filters to reduce space usage).

VI-A Data sets

Experiments have been run on both real datasets and artificial ones. We have used most of the datasets of the Frequent Itemset Mining Implementations (FIMI) Repository333http://fimi.cs.helsinki.fi/. In addition, we have created three data sets based on the internet movie database (IMDB).

Fig. 7 contains some key figures on the data sets.

VI-B Results and discussion

Fig. 8 shows the results of our experiments for the cosine measure. The time and space for BiSam is a random variable. The reported number is an exact computation of the expectation of this random variable. Separate experiments have confirmed that observed time and space is relatively well concentrated around this value. The value of $\Delta$ used is also shown — it was chosen manually in each case to give a “human readable” output of around 1000 pairs. (For the IMDB data sets and the Kosarak data set this was not possible; for the latter this behaviour was due to a large number of false positives.) Note that choosing a smaller $\Delta$ would bring the performance of BiSam closer to the exact algorithms; this is not surprising, since lowering $\Delta$ means reporting pairs having a smaller similarity measure, increasing in this way the number of samples taken. As noted before, we are usually interested in reporting pairs with high similarity, for almost any reasonable scenario.

The results for the other measures are omitted for space reasons, since they are very similar to the ones reported here. This is because the complexity of BiSam is, in most cases, dominated by the first phase (counting item frequencies), meaning that fluctuations in the cost of the second phase have little effect. This also suggests that we could increase the value of $\mu$ (and possibly increase the value of the threshold $\mu/2$ used in the BiSam algorithm) without significantly changing the time complexity of the algorithm.

We see that the speedup obtained in the experiments varies between a factor 2 and a factor over 30. Figures 9(a) and 9(b) give a graphical overview. The largest speedups tend to come for data sets with the largest average transaction size, or data sets where some transactions are very large (e.g. Kosarak). However, as our theoretical analysis suggests, large transaction size alone is not sufficient to ensure a large speedup — items also need to have support that is not too small. So while the DirectorActor data set has very large average transaction size, the speedup is only moderate because the support of items is low. In a nutshell, BiSam gives the largest speedups when there is a combination of relatively large transactions and relatively high average support. The space usage of BiSam ranges from being quite close to the space usage for exact counting, to a decent reduction.

Though we have not experimented with methods based on locality-sensitive hashing (LSH), we observe that our method appears to have an advantage when the number $n$ of distinct items is large. This is because LSH in general (and in particular for cosine similarity) requires comparison of $\binom{n}{2}$ pairs of hash signatures. For the data sets Retail, BMS-Webview-2, Actors, and MovieActors the ratio between the number of signature comparisons and the number of hash table operations required for BiSam is in the range 9–265. While these numbers are not necessarily directly comparable, it does indicate that BiSam has the potential to improve LSH-based methods that require comparison of all signature pairs.

VII Conclusion

We have presented a new sampling-based method for finding associations in data. Besides our initial experiments, indicating that large speedups may be obtained, there appear to be many opportunities for using our approach to implement association mining systems with very high performance. Some such opportunities are outlined in Section V, but many nontrivial aspects would have to be considered to do this in the best way.

Acknowledgment

We wish to thank Blue Martini Software for contributing the KDD Cup 2000 data. Also, we thank the reviewers of the ICDM submission for pointing out several related works.