PathFinder: Efficiently Supporting Conjunctions and Disjunctions for Filtered Approximate Nearest Neighbor Search

TL;DR

PathFinder is a novel indexing framework that efficiently supports complex filtered approximate nearest neighbor searches by combining attribute-specific indexes with a cost-based optimizer, significantly improving query throughput.

Contribution

It introduces a new optimization metric, a two-phase optimization process, and an index borrowing technique for complex filtered ANNS, addressing limitations of prior graph-based indexes.

Findings

Outperforms baselines by up to 9.8x in throughput

Achieves high recall of 0.95 in experiments

Supports complex conjunctions and disjunctions efficiently

Abstract

Filtered approximate nearest neighbor search (ANNS) restricts the search to data objects whose attributes satisfy a given filter and retrieves the top-$K$ objects that are most semantically similar to the query object. Many graph-based ANNS indexes are proposed to enable efficient filtered ANNS but remain limited in applicability or performance: indexes optimized for a specific attribute achieve high efficiency for filters on that attribute but fail to support complex filters with arbitrary conjunctions and disjunctions over multiple attributes. Inspired by the design of relational databases, this paper presents PathFinder, a new indexing framework that allows users to selectively create ANNS indexes optimized for filters on specific attributes and employs a cost-based optimizer to efficiently utilize them for processing complex filters. PathFinder includes three novel techniques: 1) a new optimization metric that captures the tradeoff between query execution time and accuracy, 2) a two-phase optimization for handling filters with conjunctions and disjunctions, and 3) an index borrowing optimization that uses an attribute-specific index to process filters on another attribute. Experiments on four real-world datasets show that PathFinder outperforms the best baseline by up to 9.8x in query throughput at recall 0.95.