PiPNN: Ultra-Scalable Graph-Based Nearest Neighbor Indexing

TL;DR

PiPNN introduces a novel scalable graph construction algorithm for Approximate Nearest Neighbor Search that significantly reduces index building time while maintaining high query performance, enabling billion-scale datasets to be indexed rapidly.

Contribution

PiPNN's HashPrune algorithm enables ultra-fast, scalable graph-based index construction without large memory overhead, outperforming existing methods in speed and scalability.

Findings

PiPNN is up to 12.9x faster than HNSW and Vamana.

Builds high-quality indexes on billion-scale datasets in under 20 minutes.

Achieves higher query throughput than previous algorithms.

Abstract

The fastest indexes for Approximate Nearest Neighbor Search today are also the slowest to build: graph-based methods like HNSW and Vamana achieve state-of-the-art query performance but have large construction times due to relying on random-access-heavy beam searches. We introduce PiPNN (Pick-in-Partitions Nearest Neighbors), an ultra-scalable graph construction algorithm that avoids this ``search bottleneck'' that existing graph-based methods suffer from. PiPNN's core innovation is HashPrune, a novel online pruning algorithm which dynamically maintains sparse collections of edges. HashPrune enables PiPNN to partition the dataset into overlapping sub-problems, efficiently perform bulk distance comparisons via dense matrix multiplication kernels, and stream a subset of the edges into HashPrune. HashPrune guarantees bounded memory during index construction which permits PiPNN to build higher quality indices without the use of extra intermediate memory. PiPNN builds state-of-the-art indexes up to 11.6x faster than Vamana (DiskANN) and up to 12.9x faster than HNSW. PiPNN is significantly more scalable than recent algorithms for fast graph construction. PiPNN builds indexes at least 19.1x faster than MIRAGE and 17.3x than FastKCNA while producing indexes that achieve higher query throughput. PiPNN enables us to build, for the first time, high-quality ANN indexes on billion-scale datasets in under 20 minutes using a single multicore machine.