Fast Tuning the Index Construction Parameters of Proximity Graphs in Vector Databases

TL;DR

This paper introduces FastPGT, a framework that significantly accelerates the tuning of proximity graph parameters for high-dimensional vector search, enabling faster and efficient index construction without sacrificing performance.

Contribution

FastPGT proposes a novel method to efficiently tune PG construction parameters by building multiple graphs simultaneously, reducing computational costs in high-dimensional spaces.

Findings

FastPGT achieves up to 2.37x speedup over VDTuner.

The method maintains high tuning quality while reducing computation.

Extensive experiments validate the efficiency and effectiveness of FastPGT.

Abstract

k-approximate nearest neighbor search (k-ANNS) in high-dimensional vector spaces is a fundamental problem across many fields. With the advent of vector databases and retrieval-augmented generation, k-ANNS has garnered increasing attention. Among existing methods, proximity graphs (PG) based approaches are the state-of-the-art (SOTA) methods. However, the construction parameters of PGs significantly impact their search performance. Before constructing a PG for a given dataset, it is essential to tune these parameters, which first recommends a set of promising parameters and then estimates the quality of each parameter by building the corresponding PG and then testing its k-ANNS performance. Given that the construction complexity of PGs is superlinear, building and evaluating graph indexes accounts for the primary cost of parameter tuning. Unfortunately, there is currently no method considered and optimized this process.In this paper, we introduce FastPGT, an efficient framework for tuning the PG construction parameters. FastPGT accelerates parameter estimation by building multiple PGs simultaneously, thereby reducing repeated computations. Moreover, we modify the SOTA tuning model to recommend multiple parameters at once, which can be efficiently estimated using our method of building multiple PGs simultaneously. Through extensive experiments on real-world datasets, we demonstrate that FastPGT achieves up to 2.37x speedup over the SOTA method VDTuner, without compromising tuning quality.