Fast k-Nearest Neighbour Search via Dynamic Continuous Indexing

TL;DR

This paper introduces a novel randomized algorithm for fast k-nearest neighbor search that avoids space partitioning, adapts to data density, supports dynamic updates, and outperforms existing methods like LSH in accuracy and efficiency.

Contribution

A new dynamic continuous indexing algorithm that improves k-NN search by avoiding space partitioning and providing better speed, accuracy, and adaptability.

Findings

Outperforms LSH in approximation quality

Runs in linear time relative to data dimensionality

Supports dynamic dataset updates

Abstract

Existing methods for retrieving k-nearest neighbours suffer from the curse of dimensionality. We argue this is caused in part by inherent deficiencies of space partitioning, which is the underlying strategy used by most existing methods. We devise a new strategy that avoids partitioning the vector space and present a novel randomized algorithm that runs in time linear in dimensionality of the space and sub-linear in the intrinsic dimensionality and the size of the dataset and takes space constant in dimensionality of the space and linear in the size of the dataset. The proposed algorithm allows fine-grained control over accuracy and speed on a per-query basis, automatically adapts to variations in data density, supports dynamic updates to the dataset and is easy-to-implement. We show appealing theoretical properties and demonstrate empirically that the proposed algorithm outperforms locality-sensitivity hashing (LSH) in terms of approximation quality, speed and space efficiency.