LANNS: A Web-Scale Approximate Nearest Neighbor Lookup System

TL;DR

LANNS is a scalable, high-performance approximate nearest neighbor search system designed for web-scale datasets, outperforming existing methods in speed and capacity for large, high-dimensional data.

Contribution

The paper introduces LANNS, a new platform that scales to 180 million high-dimensional data points, enabling fast, approximate nearest neighbor searches at web scale.

Findings

Supports datasets of 180 million points in high dimensions

Achieves latency of a few milliseconds per query

Handles 2.5k queries per second on a single node

Abstract

Nearest neighbor search (NNS) has a wide range of applications in information retrieval, computer vision, machine learning, databases, and other areas. Existing state-of-the-art algorithm for nearest neighbor search, Hierarchical Navigable Small World Networks(HNSW), is unable to scale to large datasets of 100M records in high dimensions. In this paper, we propose LANNS, an end-to-end platform for Approximate Nearest Neighbor Search, which scales for web-scale datasets. Library for Large Scale Approximate Nearest Neighbor Search (LANNS) is deployed in multiple production systems for identifying topK ($100 \leq topK \leq 200$) approximate nearest neighbors with a latency of a few milliseconds per query, high throughput of 2.5k Queries Per Second (QPS) on a single node, on large ($\sim$180M data points) high dimensional (50-2048 dimensional) datasets.