VIBE: Vector Index Benchmark for Embeddings

TL;DR

VIBE is a comprehensive, up-to-date benchmarking framework for evaluating vector indexes in approximate nearest neighbor search, reflecting modern application needs and including diverse datasets for realistic performance assessment.

Contribution

Introduces VIBE, a new open-source benchmark with datasets and evaluation pipelines tailored for modern ANN applications, including out-of-distribution scenarios.

Findings

Benchmarking 21 vector indexes across diverse datasets

Reveals performance differences in in-distribution and OOD scenarios

Provides insights into the state-of-the-art ANN algorithms' strengths and weaknesses

Abstract

Approximate nearest neighbor (ANN) search is a performance-critical component of many machine learning pipelines. Rigorous benchmarking is essential for evaluating the performance of vector indexes for ANN search. However, the datasets of the existing benchmarks are no longer representative of the current applications of ANN search. Hence, there is an urgent need for an up-to-date set of benchmarks. To this end, we introduce Vector Index Benchmark for Embeddings (VIBE), an open source project for benchmarking ANN algorithms. VIBE contains a pipeline for creating benchmark datasets using dense embedding models characteristic of modern applications, such as retrieval-augmented generation (RAG). To replicate real-world workloads, we also include out-of-distribution (OOD) datasets where the queries and the corpus are drawn from different distributions. We use VIBE to conduct a comprehensive evaluation of SOTA vector indexes, benchmarking 21 implementations on 12 in-distribution and 6 out-of-distribution datasets.