Incremental IVF Index Maintenance for Streaming Vector Search
Jason Mohoney, Anil Pacaci, Shihabur Rahman Chowdhury, Umar Farooq, Minhas, Jeffery Pound, Cedric Renggli, Nima Reyhani, Ihab F. Ilyas, Theodoros, Rekatsinas, Shivaram Venkataraman

TL;DR
This paper introduces Ada-IVF, an incremental index maintenance method for vector search that improves update throughput and maintains search quality in streaming data scenarios.
Contribution
Ada-IVF provides an adaptive maintenance policy and local re-clustering mechanism for efficient incremental IVF index updates.
Findings
Achieves 2x to 5x higher update throughput than existing methods.
Maintains search quality with less costly index updates.
Effective for streaming and dynamic data workloads.
Abstract
The prevalence of vector similarity search in modern machine learning applications and the continuously changing nature of data processed by these applications necessitate efficient and effective index maintenance techniques for vector search indexes. Designed primarily for static workloads, existing vector search indexes degrade in search quality and performance as the underlying data is updated unless costly index reconstruction is performed. To address this, we introduce Ada-IVF, an incremental indexing methodology for Inverted File (IVF) indexes. Ada-IVF consists of 1) an adaptive maintenance policy that decides which index partitions are problematic for performance and should be repartitioned and 2) a local re-clustering mechanism that determines how to repartition them. Compared with state-of-the-art dynamic IVF index maintenance strategies, Ada-IVF achieves an average of 2x and…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsAlgorithms and Data Compression · Advanced Bandit Algorithms Research · Data Stream Mining Techniques
