Passing the Baton: High Throughput Distributed Disk-Based Vector Search with BatANN

TL;DR

BatANN is a novel distributed disk-based vector search system that scales efficiently across multiple servers, maintaining high recall and low latency for billion-scale datasets using standard TCP.

Contribution

Introduces BatANN, the first open-source distributed disk-based vector search system with a global graph, achieving near-linear throughput scaling and improved locality.

Findings

BatANN achieves 3.5-5.59x the scatter-gather baseline at 1B points.

It attains 1.44-2.09x the throughput of DistributedANN.

Mean latency remains below 3 ms on standard TCP.

Abstract

Vector search underpins modern information-retrieval systems, including retrieval-augmented generation (RAG) pipelines and search engines over unstructured text and images. As datasets scale to billions of vectors, disk-based vector search has emerged as a practical solution. However, looking to the future, we must anticipate datasets too large for any single server and throughput demands that exceed the limits of locally attached SSDs. We present BatANN, a distributed disk-based approximate nearest neighbor (ANN) system that retains the logarithmic search efficiency of a single global graph while achieving near-linear throughput scaling in the number of servers. Our core innovation is that when accessing a neighborhood which is stored on another machine, we send the full state of the query to the other machine to continue executing there for improved locality. On 1B-point datasets at 0.95 recall using 10 servers, BatANN achieves 3.5-5.59x of the scatter-gather baseline and 1.44-2.09x the throughput of DistributedANN, respectively, while maintaining mean latency below 3 ms. Moreover, we get these results on standard TCP. To our knowledge, BatANN is the first open-source distributed disk-based vector search system to operate over a single global graph.