PageRank Pipeline Benchmark: Proposal for a Holistic System Benchmark for Big-Data Platforms

TL;DR

This paper proposes a comprehensive, scalable PageRank pipeline benchmark for big data systems, inspired by supercomputing methodologies, to enable rigorous performance evaluation across diverse hardware and software platforms.

Contribution

It introduces a holistic, multi-kernel benchmark based on PageRank, integrating existing benchmarks and adaptable to various programming environments and system scales.

Findings

Implemented in multiple languages with measured single-threaded performance.

Demonstrates scalability in both problem size and hardware.

Provides a rigorous, mathematically defined benchmarking framework.

Abstract

The rise of big data systems has created a need for benchmarks to measure and compare the capabilities of these systems. Big data benchmarks present unique scalability challenges. The supercomputing community has wrestled with these challenges for decades and developed methodologies for creating rigorous scalable benchmarks (e.g., HPC Challenge). The proposed PageRank pipeline benchmark employs supercomputing benchmarking methodologies to create a scalable benchmark that is reflective of many real-world big data processing systems. The PageRank pipeline benchmark builds on existing prior scalable benchmarks (Graph500, Sort, and PageRank) to create a holistic benchmark with multiple integrated kernels that can be run together or independently. Each kernel is well defined mathematically and can be implemented in any programming environment. The linear algebraic nature of PageRank makes it well suited to being implemented using the GraphBLAS standard. The computations are simple enough that performance predictions can be made based on simple computing hardware models. The surrounding kernels provide the context for each kernel that allows rigorous definition of both the input and the output for each kernel. Furthermore, since the proposed PageRank pipeline benchmark is scalable in both problem size and hardware, it can be used to measure and quantitatively compare a wide range of present day and future systems. Serial implementations in C++, Python, Python with Pandas, Matlab, Octave, and Julia have been implemented and their single threaded performance has been measured.