Dynamic Similarity Search on Integer Sketches

TL;DR

This paper introduces DyFT, a dynamic filter trie method that significantly improves the speed and memory efficiency of similarity searches on integer and binary sketches, especially for large, evolving datasets.

Contribution

The paper presents DyFT, a novel dynamic similarity search algorithm optimized for integer sketches, addressing limitations of existing methods in scalability and efficiency for dynamic datasets.

Findings

DyFT outperforms existing methods in speed by 6,000 times on large datasets.

DyFT reduces memory usage to one-thirteenth of comparable techniques.

Experimental results demonstrate DyFT's superior scalability and efficiency.

Abstract

Similarity-preserving hashing is a core technique for fast similarity searches, and it randomly maps data points in a metric space to strings of discrete symbols (i.e., sketches) in the Hamming space. While traditional hashing techniques produce binary sketches, recent ones produce integer sketches for preserving various similarity measures. However, most similarity search methods are designed for binary sketches and inefficient for integer sketches. Moreover, most methods are either inapplicable or inefficient for dynamic datasets, although modern real-world datasets are updated over time. We propose dynamic filter trie (DyFT), a dynamic similarity search method for both binary and integer sketches. An extensive experimental analysis using large real-world datasets shows that DyFT performs superiorly with respect to scalability, time performance, and memory efficiency. For example, on a huge dataset of 216 million data points, DyFT performs a similarity search 6,000 times faster than a state-of-the-art method while reducing to one-thirteenth in memory.