A Memory-Efficient Distributed Algorithm for Approximate Nearest Neighbour Search with Arbitrary Distances

TL;DR

This paper presents PDASC, a distributed approximate nearest neighbor search algorithm that supports arbitrary dissimilarity functions, is memory-efficient, and scalable across distributed systems, improving accuracy and efficiency in high-dimensional, heterogeneous datasets.

Contribution

PDASC introduces a novel, clustering-based distributed index that is agnostic to distance properties, enabling flexible, memory-efficient ANN search in distributed environments.

Findings

Achieves competitive accuracy with lower per-node memory.

Supports arbitrary dissimilarity functions, including non-metric.

Provides scalable, energy-efficient ANN search without hardware acceleration.

Abstract

Approximate nearest neighbour (ANN) search has become a central task in modern data-intensive applications, particularly when operating on large, heterogeneous, or high-dimensional datasets. However, many existing ANN methods struggle in such scenarios, either because they rely on metric assumptions or because their indexing strategies are not well suited to distributed environments or to settings with constrained memory resources. This work introduces PDASC (Parametrizable Distributed Approximate Similarity Search with Clustering), a distributed ANN search algorithm whose index design simultaneously supports arbitrary dissimilarity functions and efficient deployment in distributed, storage-aware environments. PDASC builds a distributed hierarchical index based on clustering mechanisms that are agnostic to distance properties, thereby accommodating non-metric and domain-specific similarities while naturally partitioning indexing and search across multiple computing nodes, with a compact per-node memory footprint. By preserving locally informative neighbourhood structure, the proposed index mitigates practical manifestations of the curse of dimensionality in high-dimensional spaces. We analyse how the index structural parameters govern the trade-offs among recall, computational cost, and memory usage. Experimental evaluation across multiple benchmark datasets and distance functions shows that PDASC achieves competitive accuracy-efficiency trade-offs while consistently requiring lower per-node memory compared to state-of-the-art ANN methods. By avoiding reliance on specialised hardware acceleration, PDASC enables scalable and energy-efficient similarity search in heterogeneous and distributed settings where memory efficiency and distance-function flexibility are first-class constraints.