Frequency-Aware Graph Construction and Search for Dynamic Vector Databases

TL;DR

This paper introduces DQF, a dual-index framework for dynamic vector databases that adaptively manages hot and cold data, significantly improving search speed while maintaining high recall in changing query environments.

Contribution

The paper proposes a novel dual-layer index and dynamic search strategy that effectively handles fluctuating query frequencies without full index rebuilds.

Findings

Achieves 2.0-5.7x speedup over state-of-the-art methods

Maintains 95% recall rate in dynamic query scenarios

Avoids full index reconstruction despite changing query distributions

Abstract

Approximate Nearest Neighbor Search (ANNS) is a crucial operation in databases and artificial intelligence. While graph-based ANNS methods like HNSW and NSG excel in performance, they assume uniform query distribution. However, in real-world scenarios, user preferences and temporal dynamics often result in certain data points being queried more frequently than others, and these query patterns can change over time. To better leverage such characteristics, we propose DQF, a novel Dual-Index Query Framework. This framework features a dual-layer index structure and a dynamic search strategy based on a decision tree. The dual-layer index includes a hot index for high-frequency nodes and a full index covering the entire dataset, allowing for the separate management of hot and cold queries. Furthermore, we propose a dynamic search strategy that employs a decision tree to determine whether a query is of the high-frequency type, avoiding unnecessary searches in the full index through early termination. Additionally, to address fluctuations in query frequency, we design an update mechanism to manage the hot index. New high-frequency nodes will be inserted into the hot index, which is periodically rebuilt when its size exceeds a predefined threshold, removing outdated low-frequency nodes. Experiments on four real-world datasets demonstrate that the Dual-Index Query Framework achieves a significant speedup of 2.0-5.7x over state-of-the-art algorithms while maintaining a 95% recall rate. Importantly, it avoids full index reconstruction even as query distributions change, underscoring its efficiency and practicality in dynamic query distribution scenarios.