A Generic Inverted Index Framework for Similarity Search on the GPU - Technical Report

TL;DR

This paper introduces GENIE, a GPU-based generic inverted index framework that simplifies parallel similarity search across various data types, featuring novel data structures and LSH concepts, validated through extensive experiments.

Contribution

The paper presents GENIE, a flexible GPU framework for similarity search, including new data structures c-PQ and the $ au$-ANN LSH, enhancing efficiency and supporting multiple data types.

Findings

GENIE reduces programming complexity for GPU similarity search.

The framework supports several popular data types and measures.

Experimental results show high efficiency and effectiveness.

Abstract

We propose a novel generic inverted index framework on the GPU (called GENIE), aiming to reduce the programming complexity of the GPU for parallel similarity search of different data types. Not every data type and similarity measure are supported by GENIE, but many popular ones are. We present the system design of GENIE, and demonstrate similarity search with GENIE on several data types along with a theoretical analysis of search results. A new concept of locality sensitive hashing (LSH) named $\tau$-ANN search, and a novel data structure c-PQ on the GPU are also proposed for achieving this purpose. Extensive experiments on different real-life datasets demonstrate the efficiency and effectiveness of our framework. The implemented system has been released as open source.