Scalable $k$-NN graph construction

TL;DR

This paper introduces a scalable method for constructing approximate k-NN graphs that balances efficiency and accuracy, suitable for large-scale high-dimensional data, by hierarchical partitioning, multiple graph merging, and neighborhood propagation.

Contribution

It presents a novel hierarchical and randomized approach combined with neighborhood propagation for efficient and accurate large-scale k-NN graph construction.

Findings

Significant speed-up in large-scale visual data processing

Theoretical and empirical validation of accuracy and efficiency

Improved approximate k-NN graph quality over existing methods

Abstract

The $k$-NN graph has played a central role in increasingly popular data-driven techniques for various learning and vision tasks; yet, finding an efficient and effective way to construct $k$-NN graphs remains a challenge, especially for large-scale high-dimensional data. In this paper, we propose a new approach to construct approximate $k$-NN graphs with emphasis in: efficiency and accuracy. We hierarchically and randomly divide the data points into subsets and build an exact neighborhood graph over each subset, achieving a base approximate neighborhood graph; we then repeat this process for several times to generate multiple neighborhood graphs, which are combined to yield a more accurate approximate neighborhood graph. Furthermore, we propose a neighborhood propagation scheme to further enhance the accuracy. We show both theoretical and empirical accuracy and efficiency of our approach to $k$-NN graph construction and demonstrate significant speed-up in dealing with large scale visual data.