Fishing in the Stream: Similarity Search over Endless Data

TL;DR

This paper introduces Stream-LSH, a novel algorithm for similarity search over endless data streams that considers data quality and temporal factors, effectively managing unbounded data with bounded resources.

Contribution

The paper proposes Stream-LSH, a new randomized algorithm that bounds index size by prioritizing data based on freshness, quality, and popularity, improving search probability.

Findings

Stream-LSH outperforms alternative methods in finding similar items.

Empirical results confirm theoretical advantages of Stream-LSH.

The approach effectively manages unbounded data with limited resources.

Abstract

Similarity search is the task of retrieving data items that are similar to a given query. In this paper, we introduce the time-sensitive notion of similarity search over endless data-streams (SSDS), which takes into account data quality and temporal characteristics in addition to similarity. SSDS is challenging as it needs to process unbounded data, while computation resources are bounded. We propose Stream-LSH, a randomized SSDS algorithm that bounds the index size by retaining items according to their freshness, quality, and dynamic popularity attributes. We analytically show that Stream-LSH increases the probability to find similar items compared to alternative approaches using the same space capacity. We further conduct an empirical study using real world stream datasets, which confirms our theoretical results.