gSuite: A Flexible and Framework Independent Benchmark Suite for Graph Neural Network Inference on GPUs

TL;DR

gSuite is a new, flexible benchmark suite for GNN inference on GPUs that is framework-independent, supports various computational models, and facilitates detailed performance analysis using real hardware and simulators.

Contribution

It introduces gSuite, a framework-independent benchmark suite that simplifies performance characterization of GNNs across different computational models and hardware setups.

Findings

gSuite enables detailed GNN performance analysis on GPUs.

Different computational models significantly affect GNN performance.

Benchmark results highlight the impact of model choice and hardware.

Abstract

As the interest to Graph Neural Networks (GNNs) is growing, the importance of benchmarking and performance characterization studies of GNNs is increasing. So far, we have seen many studies that investigate and present the performance and computational efficiency of GNNs. However, the work done so far has been carried out using a few high-level GNN frameworks. Although these frameworks provide ease of use, they contain too many dependencies to other existing libraries. The layers of implementation details and the dependencies complicate the performance analysis of GNN models that are built on top of these frameworks, especially while using architectural simulators. Furthermore, different approaches on GNN computation are generally overlooked in prior characterization studies, and merely one of the common computational models is evaluated. Based on these shortcomings and needs that we observed, we developed a benchmark suite that is framework independent, supporting versatile computational models, easily configurable and can be used with architectural simulators without additional effort. Our benchmark suite, which we call gSuite, makes use of only hardware vendor's libraries and therefore it is independent of any other frameworks. gSuite enables performing detailed performance characterization studies on GNN Inference using both contemporary GPU profilers and architectural GPU simulators. To illustrate the benefits of our new benchmark suite, we perform a detailed characterization study with a set of well-known GNN models with various datasets; running gSuite both on a real GPU card and a timing-detailed GPU simulator. We also implicate the effect of computational models on performance. We use several evaluation metrics to rigorously measure the performance of GNN computation.