Vertical, Temporal, and Horizontal Scaling of Hierarchical Hypersparse GraphBLAS Matrices

TL;DR

This paper evaluates the performance of hierarchical hypersparse GraphBLAS matrices across various hardware platforms, demonstrating significant scalability and enabling large-scale streaming data analysis.

Contribution

It provides a comprehensive performance analysis of hierarchical hypersparse GraphBLAS matrices on diverse hardware, highlighting scalability and enabling large-scale data processing.

Findings

Single-core performance up to 4 million updates/sec

Single node performance up to 170 million updates/sec

SuperCloud achieved over 200 billion updates/sec

Abstract

Hypersparse matrices are a powerful enabler for a variety of network, health, finance, and social applications. Hierarchical hypersparse GraphBLAS matrices enable rapid streaming updates while preserving algebraic analytic power and convenience. In many contexts, the rate of these updates sets the bounds on performance. This paper explores hierarchical hypersparse update performance on a variety of hardware with identical software configurations. The high-level language bindings of the GraphBLAS readily enable performance experiments on simultaneous diverse hardware. The best single process performance measured was 4,000,000 updates per second. The best single node performance measured was 170,000,000 updates per second. The hardware used spans nearly a decade and allows a direct comparison of hardware improvements for this computation over this time range; showing a 2x increase in single-core performance, a 3x increase in single process performance, and a 5x increase in single node performance. Running on nearly 2,000 MIT SuperCloud nodes simultaneously achieved a sustained update rate of over 200,000,000,000 updates per second. Hierarchical hypersparse GraphBLAS allows the MIT SuperCloud to analyze extremely large streaming network data sets.