Filtered Approximate Nearest Neighbor Search Cost Estimation

TL;DR

This paper introduces E2E, a cost estimation framework for filtered AKNN search that improves efficiency by 2-3 times through better cost prediction and early termination strategies.

Contribution

The paper presents a novel cost estimation model that explicitly captures query and attribute correlations, enhancing filtered AKNN search optimization.

Findings

E2E improves retrieval efficiency by 2x-3x.

The model achieves higher estimation accuracy than existing methods.

Experimental results validate the effectiveness on real-world datasets.

Abstract

Hybrid queries combining high-dimensional vector similarity with structured attribute filtering have garnered significant attention across both academia and industry. A critical instance of this paradigm is filtered Approximate k Nearest Neighbor (AKNN) search, where embeddings (e.g., image or text) are queried alongside constraints such as labels or numerical range. While essential for rich retrieval, optimizing these queries remains challenging due to the highly variable search cost induced by combined filters. In this paper, we propose a novel cost estimation framework, E2E, for filtered AKNN search and demonstrate its utility in downstream optimization tasks, specifically early termination. Unlike existing approaches, our model explicitly captures the correlation between the query vector distribution and attribute-value selectivity, yielding significantly higher estimation accuracy. By leveraging these estimates to refine search termination conditions, we achieve substantial performance gains. Experimental results on real-world datasets demonstrate that our approach improves retrieval efficiency by 2x-3x over state-of-the-art baselines while maintaining high search accuracy.