Revisiting Graph Analytics Benchmark

TL;DR

This paper introduces a comprehensive graph analytics benchmark that includes carefully selected algorithms, new synthetic datasets, and an innovative LLM-based API usability evaluation framework, addressing limitations of previous benchmarks.

Contribution

It presents a novel benchmark with an expanded algorithm set, new datasets, and a unique LLM-based API usability assessment, improving evaluation comprehensiveness.

Findings

The benchmark outperforms existing platforms in evaluations.

The new datasets enhance testing diversity.

The LLM-based API evaluation provides valuable usability insights.

Abstract

The rise of graph analytics platforms has led to the development of various benchmarks for evaluating and comparing platform performance. However, existing benchmarks often fall short of fully assessing performance due to limitations in core algorithm selection, data generation processes (and the corresponding synthetic datasets), as well as the neglect of API usability evaluation. To address these shortcomings, we propose a novel graph analytics benchmark. First, we select eight core algorithms by extensively reviewing both academic and industrial settings. Second, we design an efficient and flexible data generator and produce eight new synthetic datasets as the default datasets for our benchmark. Lastly, we introduce a multi-level large language model (LLM)-based framework for API usability evaluation-the first of its kind in graph analytics benchmarks. We conduct comprehensive experimental evaluations on existing platforms (GraphX, PowerGraph, Flash, Grape, Pregel+, Ligra and G-thinker). The experimental results demonstrate the superiority of our proposed benchmark.