Exploring the Meaningfulness of Nearest Neighbor Search in High-Dimensional Space

TL;DR

This study investigates the effectiveness of nearest neighbor search in high-dimensional spaces, especially for text embeddings, revealing they are less affected by the curse of dimensionality and that distance function choice has minimal impact.

Contribution

The paper provides extensive empirical analysis of NNS in high-dimensional embeddings, demonstrating their robustness and practical relevance across various datasets and distance metrics.

Findings

High-dimensional text embeddings are more resilient to the curse of dimensionality.

The choice of distance function has minimal impact on NNS relevance.

Dense vector representations remain effective for retrieval tasks in high dimensions.

Abstract

Dense high dimensional vectors are becoming increasingly vital in fields such as computer vision, machine learning, and large language models (LLMs), serving as standard representations for multimodal data. Now the dimensionality of these vector can exceed several thousands easily. Despite the nearest neighbor search (NNS) over these dense high dimensional vectors have been widely used for retrieval augmented generation (RAG) and many other applications, the effectiveness of NNS in such a high-dimensional space remains uncertain, given the possible challenge caused by the "curse of dimensionality." To address above question, in this paper, we conduct extensive NNS studies with different distance functions, such as $L_1$ distance, $L_2$ distance and angular-distance, across diverse embedding datasets, of varied types, dimensionality and modality. Our aim is to investigate factors influencing the meaningfulness of NNS. Our experiments reveal that high-dimensional text embeddings exhibit increased resilience as dimensionality rises to higher levels when compared to random vectors. This resilience suggests that text embeddings are less affected to the "curse of dimensionality," resulting in more meaningful NNS outcomes for practical use. Additionally, the choice of distance function has minimal impact on the relevance of NNS. Our study shows the effectiveness of the embedding-based data representation method and can offer opportunity for further optimization of dense vector-related applications.