Scalable Distributed-Memory External Sorting

TL;DR

This paper presents scalable distributed-memory algorithms for external sorting on large data sets, achieving near-optimal I/O performance and outperforming existing methods in benchmark tests.

Contribution

It introduces two novel algorithms for distributed external sorting that minimize I/O and communication overhead, with one achieving near-minimal I/O and the other balancing I/O and communication.

Findings

Algorithms work with just two passes over data for large inputs

Implementation outperforms competitors in sorting benchmarks

Algorithms are based on multiway merging paradigm

Abstract

We engineer algorithms for sorting huge data sets on massively parallel machines. The algorithms are based on the multiway merging paradigm. We first outline an algorithm whose I/O requirement is close to a lower bound. Thus, in contrast to naive implementations of multiway merging and all other approaches known to us, the algorithm works with just two passes over the data even for the largest conceivable inputs. A second algorithm reduces communication overhead and uses more conventional specifications of the result at the cost of slightly increased I/O requirements. An implementation wins the well known sorting benchmark in several categories and by a large margin over its competitors.