An Empirical Study of Realized GNN Expressiveness

TL;DR

This paper introduces BREC, a challenging dataset for empirically measuring the actual expressiveness of GNNs beyond the 1-WL test, revealing gaps between theoretical and practical capabilities.

Contribution

It presents a novel dataset and comprehensive evaluation of beyond-1-WL GNN models, providing the first empirical measurement of their realized expressiveness.

Findings

BREC dataset with higher difficulty and granularity

State-of-the-art beyond-1-WL GNNs show limited realized expressiveness

Significant gap between theoretical and empirical expressiveness

Abstract

Research on the theoretical expressiveness of Graph Neural Networks (GNNs) has developed rapidly, and many methods have been proposed to enhance the expressiveness. However, most methods do not have a uniform expressiveness measure except for a few that strictly follow the $k$-dimensional Weisfeiler-Lehman ($k$-WL) test hierarchy, leading to difficulties in quantitatively comparing their expressiveness. Previous research has attempted to use datasets for measurement, but facing problems with difficulty (any model surpassing 1-WL has nearly 100% accuracy), granularity (models tend to be either 100% correct or near random guess), and scale (only several essentially different graphs involved). To address these limitations, we study the realized expressive power that a practical model instance can achieve using a novel expressiveness dataset, BREC, which poses greater difficulty (with up to 4-WL-indistinguishable graphs), finer granularity (enabling comparison of models between 1-WL and 3-WL), a larger scale (consisting of 800 1-WL-indistinguishable graphs that are non-isomorphic to each other). We synthetically test 23 models with higher-than-1-WL expressiveness on BREC. Our experiment gives the first thorough measurement of the realized expressiveness of those state-of-the-art beyond-1-WL GNN models and reveals the gap between theoretical and realized expressiveness. Dataset and evaluation codes are released at: https://github.com/GraphPKU/BREC.