GraphNet: A Large-Scale Computational Graph Dataset for Tensor Compiler Research

TL;DR

GraphNet provides a comprehensive dataset of real-world deep learning computational graphs and introduces new benchmarking metrics to evaluate tensor compiler performance across multiple frameworks and tasks.

Contribution

The paper introduces GraphNet, a large-scale dataset of deep learning graphs, and proposes the Speedup Score S(t) and Error-aware Speedup Score ES(t) for evaluating tensor compiler performance.

Findings

Benchmarking shows the performance of CINN and TorchInductor on CV and NLP tasks.

The dataset enables reliable evaluation of compiler optimization capabilities.

Tools for graph extraction and compiler evaluation are publicly available.

Abstract

We introduce GraphNet, a dataset of 2.7K real-world deep learning computational graphs with rich metadata, spanning six major task categories across multiple deep learning frameworks. To evaluate tensor compiler performance on these samples, we propose the benchmark metric Speedup Score S(t), which jointly considers runtime speedup and execution correctness under tunable tolerance levels, offering a reliable measure of general optimization capability. Furthermore, we extend S(t) to the Error-aware Speedup Score ES(t), which incorporates error information and helps compiler developers identify key performance bottlenecks. In this report, we benchmark the default tensor compilers, CINN for PaddlePaddle and TorchInductor for PyTorch, on computer vision (CV) and natural language processing (NLP) samples to demonstrate the practicality of GraphNet. The full construction pipeline with graph extraction and compiler evaluation tools is available at https://github.com/PaddlePaddle/GraphNet .