A Load-Balanced Parallel and Distributed Sorting Algorithm Implemented with PGX.D

TL;DR

This paper introduces a load-balanced distributed sorting algorithm implemented in PGX.D that significantly outperforms Spark, effectively handles duplicate data, and maintains balanced workloads across various data distributions.

Contribution

The paper presents a novel distributed sorting algorithm in PGX.D that improves performance, load balancing, and duplicate data handling compared to existing systems.

Findings

Achieves 2x-3x faster sorting than Spark

Handles datasets with many duplicates efficiently

Maintains balanced workloads across different data distributions

Abstract

Sorting has been one of the most challenging studied problems in different scientific researches. Although many techniques and algorithms have been proposed on the theory of having efficient parallel sorting implementation, however achieving desired performance on different types of the architectures with large number of processors is still a challenging issue. Maximizing parallelism level in applications can be achieved by minimizing overheads due to load imbalance and waiting time due to memory latencies. In this paper, we present a distributed sorting algorithm implemented in PGX.D, a fast distributed graph processing system, which outperforms the Spark's distributed sorting implementation by around 2x-3x by hiding communication latencies and minimizing unnecessary overheads. Furthermore, it shows that the proposed PGX.D sorting method handles dataset containing many duplicated data entries efficiently and always results in keeping balanced workloads for different input data distribution types.