WoW: A Window-to-Window Incremental Index for Range-Filtering Approximate Nearest Neighbor Search

TL;DR

This paper introduces a window graph-based index for range-filtering approximate nearest neighbor search that supports incremental updates and arbitrary range filters, achieving fast query times and efficient construction.

Contribution

It proposes a novel window graph-based RFANNS index with an insertion algorithm for incremental updates and optimized range search for arbitrary filters.

Findings

Index construction time is comparable to the most efficient index.

Query performance is 4x faster than the most efficient incremental index.

Index size is 0.4-0.5x smaller than the most query-efficient index.

Abstract

Given a hybrid dataset where every data object consists of a vector and an attribute value, for each query with a target vector and a range filter, range-filtering approximate nearest neighbor search (RFANNS) aims to retrieve the most similar vectors from the dataset and the corresponding attribute values fall in the query range. It is a fundamental function in vector database management systems and intelligent systems with embedding abilities. Dedicated indices for RFANNS accelerate query speed with an acceptable accuracy loss on nearest neighbors. However, they are still facing the challenges to be constructed incrementally and generalized to achieve superior query performance for arbitrary range filters. In this paper, we introduce a window graph-based RFANNS index. For incremental construction, we propose an insertion algorithm to add new vector-attribute pairs into hierarchical window graphs with varying window size. To handle arbitrary range filters, we optimize relevant window search for attribute filter checks and vector distance computations by range selectivity. Extensive experiments on real-world datasets show that for index construction, the indexing time is on par with the most building-efficient index, and 4.9x faster than the most query-efficient index with 0.4-0.5x smaller size; For RFANNS query, it is 4x faster than the most efficient incremental index, and matches the performance of the best statically-built index.