A Bulk-Parallel Priority Queue in External Memory with STXXL

TL;DR

This paper introduces a bulk-parallel external memory priority queue that leverages shared-memory parallelism and high-speed disk transfer, achieving near-maximal I/O bandwidth utilization and efficient sorting performance.

Contribution

It presents a novel design and implementation of a parallel external memory priority queue with two interfaces, optimizing insertions and extractions for high-performance disk-based operations.

Findings

Achieves 75% of I/O bandwidth on rotational disks.

Reaches 65% of I/O bandwidth on SSDs.

Performs comparably to external sorting when computation-bound.

Abstract

We propose the design and an implementation of a bulk-parallel external memory priority queue to take advantage of both shared-memory parallelism and high external memory transfer speeds to parallel disks. To achieve higher performance by decoupling item insertions and extractions, we offer two parallelization interfaces: one using "bulk" sequences, the other by defining "limit" items. In the design, we discuss how to parallelize insertions using multiple heaps, and how to calculate a dynamic prediction sequence to prefetch blocks and apply parallel multiway merge for extraction. Our experimental results show that in the selected benchmarks the priority queue reaches 75% of the full parallel I/O bandwidth of rotational disks and and 65% of SSDs, or the speed of sorting in external memory when bounded by computation.