Approximate Nearest Neighbor Search of Large Scale Vectors on Distributed Storage

TL;DR

This paper introduces DSANN, a novel distributed storage system for approximate nearest neighbor search that efficiently indexes and searches billion-scale vector datasets, overcoming storage and scalability limitations of existing methods.

Contribution

The paper presents a new graph-cluster hybrid indexing system, DSANN, supporting large-scale distributed vector search with high availability and improved efficiency.

Findings

DSANN efficiently indexes billion-scale vectors in distributed storage.

DSANN reduces index construction complexity with concurrent methods.

Experimental results show DSANN's high performance in large-scale vector search.

Abstract

Approximate Nearest Neighbor Search (ANNS) in high-dimensional space is an essential operator in many online services, such as information retrieval and recommendation. Indices constructed by the state-of-the-art ANNS algorithms must be stored in single machine's memory or disk for high recall rate and throughput, suffering from substantial storage cost, constraint of limited scale and single point of failure. While distributed storage can provide a cost-effective and robust solution, there is no efficient and effective algorithms for indexing vectors in distributed storage scenarios. In this paper, we present a new graph-cluster hybrid indexing and search system which supports Distributed Storage Approximate Nearest Neighbor Search, called DSANN. DSANN can efficiently index, store, search billion-scale vector database in distributed storage and guarantee the high availability of index service. DSANN employs the concurrent index construction method to significantly reduces the complexity of index building. Then, DSANN applies Point Aggregation Graph to leverage the structural information of graph to aggregate similar vectors, optimizing storage efficiency and improving query throughput via asynchronous I/O in distributed storage. Through extensive experiments, we demonstrate DSANN can efficiently and effectively index, store and search large-scale vector datasets in distributed storage scenarios.