Three Algorithms for Merging Hierarchical Navigable Small World Graphs

TL;DR

This paper introduces three algorithms for merging HNSW graphs, improving efficiency in distributed and incremental systems, with experimental results showing significant reductions in computational costs while maintaining accuracy.

Contribution

The paper proposes three novel algorithms for merging HNSW graphs, addressing a key challenge in distributed and incremental indexing systems.

Findings

IGTM and CGTM reduce computations by up to 70%

IGTM outperforms CGTM in efficiency

Algorithms enable efficient index consolidation in vector databases

Abstract

This paper addresses the challenge of merging hierarchical navigable small world (HNSW) graphs, a critical operation for distributed systems, incremental indexing, and database compaction. We propose three algorithms for this task: Naive Graph Merge (NGM), Intra Graph Traversal Merge (IGTM), and Cross Graph Traversal Merge (CGTM). These algorithms differ in their approach to vertex selection and candidate collection during the merge process. We conceptualize graph merging as an iterative process with four key steps: processing vertex selection, candidate collection, neighborhood construction, and information propagation. Our experimental evaluation on the SIFT1M dataset demonstrates that IGTM and CGTM significantly reduce computational costs compared to naive approaches, requiring up to 70\% fewer distance computations while maintaining comparable search accuracy. Surprisingly, IGTM outperforms CGTM in efficiency, contrary to our initial expectations. The proposed algorithms enable efficient consolidation of separately constructed indices, supporting critical operations in modern vector databases and retrieval systems that rely on HNSW for similarity search.