Quantization for Vector Search under Streaming Updates

TL;DR

This paper introduces a new dynamic quantization method for large-scale vector search that maintains high accuracy and efficiency under streaming data updates, avoiding costly index rebuilds.

Contribution

It provides a theoretical framework and a practical algorithm for data-dependent quantization that remains consistent and accurate with streaming dataset modifications.

Findings

Outperforms baseline methods in large-scale streaming nearest neighbor search

Proves bounded disk I/O per update maintains accuracy guarantees

Develops a practical, adaptive quantization algorithm

Abstract

Large-scale vector databases for approximate nearest neighbor (ANN) search typically store a quantized dataset in main memory for fast access, and full precision data on remote disk. State-of-the-art ANN quantization methods are highly data-dependent, rendering them unable to handle point insertions and deletions. This either leads to degraded search quality over time, or forces costly global rebuilds of the entire search index. In this paper, we formally study data-dependent quantization under streaming dataset updates. We formulate a computation model of limited remote disk access and define a dynamic consistency property that guarantees freshness under updates. We use it to obtain the following results: Theoretically, we prove that static data-dependent quantization can be made dynamic with bounded disk I/O per update while retaining formal accuracy guarantees for ANN search. Algorithmically, we develop a practical data-dependent quantization method which is provably dynamically consistent, adapting itself to the dataset as it evolves over time. Our experiments show that the method outperforms baselines in large-scale nearest neighbor search quantization under streaming updates.