Parallel Batch-Dynamic $k$d-Trees

TL;DR

This paper introduces BDL-trees, a parallel batch-dynamic $k$d-tree data structure that efficiently supports neighborhood queries and batch updates in dynamic, large-scale datasets, with significant speedups on multicore systems.

Contribution

We present BDL-trees, a novel parallel, batch-dynamic $k$d-tree implementation with polylogarithmic depth and optimized multicore construction, enabling efficient neighborhood queries over dynamic data.

Findings

Achieves up to 34.8x speedup for batch insertions

Achieves up to 35.5x speedup for batch deletions

Achieves up to 46.1x speedup for $k$-NN queries

Abstract

$k$d-trees are widely used in parallel databases to support efficient neighborhood/similarity queries. Supporting parallel updates to $k$d-trees is therefore an important operation. In this paper, we present BDL-tree, a parallel, batch-dynamic implementation of a $k$d-tree that allows for efficient parallel $k$-NN queries over dynamically changing point sets. BDL-trees consist of a log-structured set of $k$d-trees which can be used to efficiently insert or delete batches of points in parallel with polylogarithmic depth. Specifically, given a BDL-tree with $n$ points, each batch of $B$ updates takes $O(B\log^2{(n+B)})$ amortized work and $O(\log(n+B)\log\log{(n+B)})$ depth (parallel time). We provide an optimized multicore implementation of BDL-trees. Our optimizations include parallel cache-oblivious $k$d-tree construction and parallel bloom filter construction. Our experiments on a 36-core machine with two-way hyper-threading using a variety of synthetic and real-world datasets show that our implementation of BDL-tree achieves a self-relative speedup of up to $34.8\times$ ($28.4\times$ on average) for batch insertions, up to $35.5\times$ ($27.2\times$ on average) for batch deletions, and up to $46.1\times$ ($40.0\times$ on average) for $k$-nearest neighbor queries. In addition, it achieves throughputs of up to 14.5 million updates/second for batch-parallel updates and 6.7 million queries/second for $k$-NN queries. We compare to two baseline $k$d-tree implementations and demonstrate that BDL-trees achieve a good tradeoff between the two baseline options for implementing batch updates.