Bigger Buffer k-d Trees on Multi-Many-Core Systems

TL;DR

This paper extends buffer k-d trees to handle massive datasets across multiple devices, enabling efficient parallel nearest neighbor searches on large-scale data in astronomy.

Contribution

The authors modify buffer k-d trees and their workflow to support multi-device environments, facilitating scalable nearest neighbor searches on very large datasets.

Findings

Effective multi-device buffer k-d tree framework demonstrated

Significant speed-ups in astronomy data processing

Scalable approach for massive data sets

Abstract

A buffer k-d tree is a k-d tree variant for massively-parallel nearest neighbor search. While providing valuable speed-ups on modern many-core devices in case both a large number of reference and query points are given, buffer k-d trees are limited by the amount of points that can fit on a single device. In this work, we show how to modify the original data structure and the associated workflow to make the overall approach capable of dealing with massive data sets. We further provide a simple yet efficient way of using multiple devices given in a single workstation. The applicability of the modified framework is demonstrated in the context of astronomy, a field that is faced with huge amounts of data.