A Fair Comparison of Graph Neural Networks for Graph Classification

TL;DR

This paper critically evaluates the reproducibility of graph neural network models for classification by conducting extensive experiments, revealing insights about the importance of experimental rigor and the role of structural information.

Contribution

It provides a comprehensive, controlled comparison of popular GNN models and highlights issues in experimental practices within graph classification research.

Findings

Structural information is underutilized in some datasets.

Rigorous experimental procedures are essential for fair model comparison.

Many existing results may lack reproducibility due to methodological inconsistencies.

Abstract

Experimental reproducibility and replicability are critical topics in machine learning. Authors have often raised concerns about their lack in scientific publications to improve the quality of the field. Recently, the graph representation learning field has attracted the attention of a wide research community, which resulted in a large stream of works. As such, several Graph Neural Network models have been developed to effectively tackle graph classification. However, experimental procedures often lack rigorousness and are hardly reproducible. Motivated by this, we provide an overview of common practices that should be avoided to fairly compare with the state of the art. To counter this troubling trend, we ran more than 47000 experiments in a controlled and uniform framework to re-evaluate five popular models across nine common benchmarks. Moreover, by comparing GNNs with structure-agnostic baselines we provide convincing evidence that, on some datasets, structural information has not been exploited yet. We believe that this work can contribute to the development of the graph learning field, by providing a much needed grounding for rigorous evaluations of graph classification models.