SHINE: A Scalable HNSW Index in Disaggregated Memory

TL;DR

This paper introduces SHINE, a scalable distributed HNSW index designed for disaggregated memory architectures, achieving high accuracy and efficiency in approximate nearest neighbor search across large datasets.

Contribution

SHINE presents a novel graph-preserving distributed HNSW index that maintains accuracy while overcoming bandwidth limitations with an innovative caching strategy.

Findings

Achieves same accuracy as single-machine HNSW in distributed setting

Employs an efficient cache combining mechanism for scalability

Demonstrates high efficiency and scalability in evaluations

Abstract

Approximate nearest neighbor (ANN) search is a fundamental problem in computer science for which in-memory graph-based methods, such as Hierarchical Navigable Small World (HNSW), perform exceptionally well. To scale beyond billions of high-dimensional vectors, the index must be distributed. The disaggregated memory architecture physically separates compute and memory into two distinct hardware units and has become popular in modern data centers. Both units are connected via RDMA networks that allow compute nodes to directly access remote memory and perform all the computations, posing unique challenges for disaggregated indexes. In this work, we propose a scalable HNSW index for ANN search in disaggregated memory. In contrast to existing distributed approaches, which partition the graph at the cost of accuracy, our method builds a graph-preserving index that reaches the same accuracy as a single-machine HNSW. Continuously fetching high-dimensional vector data from remote memory leads to severe network bandwidth limitations, which we overcome by employing an efficient caching mechanism. Since answering a single query involves processing numerous unique graph nodes, caching alone is not sufficient to achieve high scalability. We logically combine the caches of the compute nodes to increase the overall cache effectiveness and confirm the efficiency and scalability of our method in our evaluation.