Approximate Vector Set Search Inspired by Fly Olfactory Neural System

TL;DR

This paper introduces BioVSS, an efficient vector set search algorithm inspired by fly olfactory systems, achieving significant speedups and high recall on large datasets by using vector quantization and Bloom filter indexing.

Contribution

BioVSS is the first to simulate fly olfactory circuits for vector set search, combining quantization and Bloom filter indexing for efficiency and accuracy.

Findings

Over 50x speedup over linear scan on million-scale datasets

Recall rate of up to 98.9%

Effective handling of high-dimensional vector sets

Abstract

Vector set search, an underexplored similarity search paradigm, aims to find vector sets similar to a query set. This search paradigm leverages the inherent structural alignment between sets and real-world entities to model more fine-grained and consistent relationships for diverse applications. This task, however, faces more severe efficiency challenges than traditional single-vector search due to the combinatorial explosion of pairings in set-to-set comparisons. In this work, we aim to address the efficiency challenges posed by the combinatorial explosion in vector set search, as well as the curse of dimensionality inherited from single-vector search. To tackle these challenges, we present an efficient algorithm for vector set search, BioVSS (Bio-inspired Vector Set Search). BioVSS simulates the fly olfactory circuit to quantize vectors into sparse binary codes and then designs an index based on the set membership property of the Bloom filter. The quantization and indexing strategy enables BioVSS to efficiently perform vector set search by pruning the search space. Experimental results demonstrate over 50 times speedup compared to linear scanning on million-scale datasets while maintaining a high recall rate of up to 98.9%, making it an efficient solution for vector set search.