The Role of Local Intrinsic Dimensionality in Benchmarking Nearest Neighbor Search

TL;DR

This paper explores how local intrinsic dimensionality (LID) influences the difficulty and performance of nearest neighbor search benchmarks, providing new visualization tools and insights into dataset diversity.

Contribution

It introduces visualization methods and empirical analysis demonstrating the impact of LID on benchmarking and highlights the lack of diversity in commonly used datasets.

Findings

LID can be used to generate query sets of varying difficulty.

Different LID distributions significantly affect search performance.

Real-world datasets are less diverse than assumed, with results generalizing across datasets.

Abstract

This paper reconsiders common benchmarking approaches to nearest neighbor search. It is shown that the concept of local intrinsic dimensionality (LID) allows to choose query sets of a wide range of difficulty for real-world datasets. Moreover, the effect of different LID distributions on the running time performance of implementations is empirically studied. To this end, different visualization concepts are introduced that allow to get a more fine-grained overview of the inner workings of nearest neighbor search principles. The paper closes with remarks about the diversity of datasets commonly used for nearest neighbor search benchmarking. It is shown that such real-world datasets are not diverse: results on a single dataset predict results on all other datasets well.