Efficient K-Nearest Neighbor Join Algorithms for High Dimensional Sparse Data

TL;DR

This paper introduces three novel algorithms for efficiently performing K-Nearest Neighbor joins specifically tailored for high dimensional sparse data, addressing a gap in existing low-dimensional-focused methods.

Contribution

It proposes three new algorithms—BF, IIB, and IIIB—for high dimensional sparse data KNN join, with extensive experimental validation.

Findings

The algorithms outperform existing methods on synthetic datasets.

The inverted index-based algorithms improve efficiency in real-world data.

Experimental results confirm effectiveness for high dimensional sparse data.

Abstract

The K-Nearest Neighbor (KNN) join is an expensive but important operation in many data mining algorithms. Several recent applications need to perform KNN join for high dimensional sparse data. Unfortunately, all existing KNN join algorithms are designed for low dimensional data. To fulfill this void, we investigate the KNN join problem for high dimensional sparse data. In this paper, we propose three KNN join algorithms: a brute force (BF) algorithm, an inverted index-based(IIB) algorithm and an improved inverted index-based(IIIB) algorithm. Extensive experiments on both synthetic and real-world datasets were conducted to demonstrate the effectiveness of our algorithms for high dimensional sparse data.