GraphChallenge.org: Raising the Bar on Graph Analytic Performance

TL;DR

The paper discusses the Graph Challenge initiative, which benchmarks and compares graph analytic systems' performance, highlighting recent innovations and establishing current state-of-the-art capabilities in processing large graphs efficiently.

Contribution

It introduces a community-driven benchmarking platform with standardized datasets, algorithms, and metrics, enabling consistent performance evaluation of graph analysis systems.

Findings

Performance scales with number of edges, roughly proportional to N_e^{4/3}

Current systems process about 10^8 edges per second for large graphs

Submissions are 30 times faster than serial implementations

Abstract

The rise of graph analytic systems has created a need for new ways to measure and compare the capabilities of graph processing systems. The MIT/Amazon/IEEE Graph Challenge has been developed to provide a well-defined community venue for stimulating research and highlighting innovations in graph analysis software, hardware, algorithms, and systems. GraphChallenge.org provides a wide range of pre-parsed graph data sets, graph generators, mathematically defined graph algorithms, example serial implementations in a variety of languages, and specific metrics for measuring performance. Graph Challenge 2017 received 22 submissions by 111 authors from 36 organizations. The submissions highlighted graph analytic innovations in hardware, software, algorithms, systems, and visualization. These submissions produced many comparable performance measurements that can be used for assessing the current state of the art of the field. There were numerous submissions that implemented the triangle counting challenge and resulted in over 350 distinct measurements. Analysis of these submissions show that their execution time is a strong function of the number of edges in the graph, $N_e$, and is typically proportional to $N_e^{4/3}$ for large values of $N_e$. Combining the model fits of the submissions presents a picture of the current state of the art of graph analysis, which is typically $10^8$ edges processed per second for graphs with $10^8$ edges. These results are $30$ times faster than serial implementations commonly used by many graph analysts and underscore the importance of making these performance benefits available to the broader community. Graph Challenge provides a clear picture of current graph analysis systems and underscores the need for new innovations to achieve high performance on very large graphs.