Benchmarking Filtered Approximate Nearest Neighbor Search Algorithms on Transformer-based Embedding Vectors

TL;DR

This paper introduces a new dataset of transformer-based embeddings with attributes for arXiv papers and benchmarks eleven FANNS algorithms to guide method selection.

Contribution

It provides the first large-scale dataset with rich attributes for transformer embeddings and a comprehensive benchmark of FANNS algorithms on this dataset.

Findings

Identified a lack of real-world attribute datasets for transformer embeddings.

Benchmark results reveal performance differences among FANNS methods.

Guidelines for selecting FANNS methods based on use case scenarios.

Abstract

Advances in embedding models for text, image, audio, and video drive progress across multiple domains, including retrieval-augmented generation, recommendation systems, and others. Many of these applications require an efficient method to retrieve items that are close to a given query in the embedding space while satisfying a filter condition based on the item's attributes, a problem known as filtered approximate nearest neighbor search (FANNS). By performing an in-depth literature analysis on FANNS, we identify a key gap in the research landscape: publicly available datasets with embedding vectors from state-of-the-art transformer-based text embedding models that contain abundant real-world attributes covering a broad spectrum of attribute types and value distributions. To fill this gap, we introduce the arxiv-for-fanns dataset of transformer-based embedding vectors for the abstracts of over 2.7 million arXiv papers, enriched with 11 real-world attributes such as authors and categories. We benchmark eleven different FANNS methods on our new dataset to evaluate their performance across different filter types, numbers of retrieved neighbors, dataset scales, and query selectivities. We distill our findings into eight key observations that guide users in selecting the most suitable FANNS method for their specific use cases.