SIEVE: Effective Filtered Vector Search with Collection of Indexes

TL;DR

SIEVE introduces a collection of specialized indexes for filtered vector search, enabling efficient retrieval with hard predicates by workload-aware index selection, outperforming existing methods in speed and resource usage.

Contribution

The paper presents a novel collection-based indexing approach for filtered vector search, with an analytical model for index selection and construction, improving efficiency and flexibility.

Findings

Achieves up to 8.06x speedup over existing methods.

Uses as low as 1% build time compared to other indexes.

Requires less than 2.15x memory of standard HNSW graph.

Abstract

Many real-world tasks such as recommending videos with the kids tag can be reduced to finding most similar vectors associated with hard predicates. This task, filtered vector search, is challenging as prior state-of-the-art graph-based (unfiltered) similarity search techniques quickly degenerate when hard constraints are considered. That is, effective graph-based filtered similarity search relies on sufficient connectivity for reaching the most similar items within just a few hops. To consider predicates, recent works propose modifying graph traversal to visit only the items that may satisfy predicates. However, they fail to offer the just-a-few-hops property for a wide range of predicates: they must restrict predicates significantly or lose efficiency if only a small fraction of items satisfy predicates. We propose an opposite approach: instead of constraining traversal, we build many indexes each serving different predicate forms. For effective construction, we devise a three-dimensional analytical model capturing relationships among index size, search time, and recall, with which we follow a workload-aware approach to pack as many useful indexes as possible into a collection. At query time, the analytical model is employed yet again to discern the one that offers the fastest search at a given recall. We show superior performance and support on datasets with varying selectivities and forms: our approach achieves up to 8.06x speedup while having as low as 1% build time versus other indexes, with less than 2.15x memory of a standard HNSW graph and modest knowledge of past workloads.