SWOOP: Top-k Similarity Joins over Set Streams

TL;DR

SWOOP is a scalable algorithm designed to efficiently maintain the top-$k$ most similar set pairs in rapid data streams, such as Twitter, by using novel indexing and filtering techniques to handle dynamic updates.

Contribution

The paper introduces SWOOP, a new stream join algorithm that efficiently maintains top-$k$ similar set pairs in high-speed data streams with minimal memory usage.

Findings

SWOOP outperforms existing methods by handling much faster stream rates.

It uses novel indexing and filtering to prune useless pairs efficiently.

SWOOP maintains a minimal stock of similar pairs for quick updates.

Abstract

We provide efficient support for applications that aim to continuously find pairs of similar sets in rapid streams of sets. A prototypical example setting is that of tweets. A tweet is a set of words, and Twitter emits about half a billion tweets per day. Our solution makes it possible to efficiently maintain the top-$k$ most similar tweets from a pair of rapid Twitter streams, e.g., to discover similar trends in two cities if the streams concern cities. Using a sliding window model, the top-$k$ result changes as new sets in the stream enter the window or existing ones leave the window. Maintaining the top-$k$ result under rapid streams is challenging. First, when a set arrives, it may form a new pair for the top-$k$ result with any set already in the window. Second, when a set leaves the window, all its pairings in the top-$k$ are invalidated and must be replaced. It is not enough to maintain the $k$ most similar pairs, as less similar pairs may eventually be promoted to the top-$k$ result. A straightforward solution that pairs every new set with all sets in the window and keeps all pairs for maintaining the top-$k$ result is memory intensive and too slow. We propose SWOOP, a highly scalable stream join algorithm that solves these issues. Novel indexing techniques and sophisticated filters efficiently prune useless pairs as new sets enter the window. SWOOP incrementally maintains a stock of similar pairs to update the top-$k$ result at any time, and the stock is shown to be minimal. Our experiments confirm that SWOOP can deal with stream rates that are orders of magnitude faster than the rates of existing approaches.