Enhancing HNSW Index for Real-Time Updates: Addressing Unreachable Points and Performance Degradation

TL;DR

This paper introduces the MN-RU algorithm to enhance HNSW index performance for real-time updates by addressing unreachable points and degradation issues, ensuring better accuracy and efficiency in dynamic data environments.

Contribution

The paper proposes the MN-RU algorithm that improves HNSW's update efficiency and reduces unreachable points, addressing long-term performance issues in dynamic datasets.

Findings

MN-RU outperforms existing methods in update efficiency

Reduces growth of unreachable points during updates

Maintains search accuracy over long-term dynamic operations

Abstract

The approximate nearest neighbor search (ANNS) is a fundamental and essential component in data mining and information retrieval, with graph-based methodologies demonstrating superior performance compared to alternative approaches. Extensive research efforts have been dedicated to improving search efficiency by developing various graph-based indices, such as HNSW (Hierarchical Navigable Small World). However, the performance of HNSW and most graph-based indices become unacceptable when faced with a large number of real-time deletions, insertions, and updates. Furthermore, during update operations, HNSW can result in some data points becoming unreachable, a situation we refer to as the `unreachable points phenomenon'. This phenomenon could significantly affect the search accuracy of the graph in certain situations. To address these issues, we present efficient measures to overcome the shortcomings of HNSW, specifically addressing poor performance over long periods of delete and update operations and resolving the issues caused by the unreachable points phenomenon. Our proposed MN-RU algorithm effectively improves update efficiency and suppresses the growth rate of unreachable points, ensuring better overall performance and maintaining the integrity of the graph. Our results demonstrate that our methods outperform existing approaches. Furthermore, since our methods are based on HNSW, they can be easily integrated with existing indices widely used in the industrial field, making them practical for future real-world applications. Code is available at \url{https://github.com/xwt1/MN-RU.git}