Scalable Overload-Aware Graph-Based Index Construction for 10-Billion-Scale Vector Similarity Search

TL;DR

This paper presents SOGAIC, a scalable, overload-aware graph construction system for billion-scale vector similarity search, significantly improving construction speed and scalability for industrial applications.

Contribution

Introduction of SOGAIC, a novel overload-aware, distributed graph construction method with adaptive partitioning and load balancing for ultra-large vector databases.

Findings

47.3% reduction in construction time

Successfully deployed in a real-world search engine

Handles over 10 billion vectors daily

Abstract

Approximate Nearest Neighbor Search (ANNS) is essential for modern data-driven applications that require efficient retrieval of top-k results from massive vector databases. Although existing graph-based ANNS algorithms achieve a high recall rate on billion-scale datasets, their slow construction speed and limited scalability hinder their applicability to large-scale industrial scenarios. In this paper, we introduce SOGAIC, the first Scalable Overload-Aware Graph-Based ANNS Index Construction system tailored for ultra-large-scale vector databases: 1) We propose a dynamic data partitioning algorithm with overload constraints that adaptively introduces overlaps among subsets; 2) To enable efficient distributed subgraph construction, we employ a load-balancing task scheduling framework combined with an agglomerative merging strategy; 3) Extensive experiments on various datasets demonstrate a reduction of 47.3% in average construction time compared to existing methods. The proposed method has also been successfully deployed in a real-world industrial search engine, managing over 10 billion daily updated vectors and serving hundreds of millions of users.