Property-Driven Evaluation of GNN Expressiveness at Scale: Datasets, Framework, and Study

TL;DR

This paper introduces a formal, property-driven evaluation framework for GNNs, using large-scale datasets and systematic metrics to analyze their ability to capture fundamental graph properties, revealing key trade-offs among pooling methods.

Contribution

It develops a comprehensive, formal evaluation methodology and large datasets for assessing GNN expressiveness across multiple properties, providing new insights into pooling method performance.

Findings

Attention-based pooling improves generalization and robustness.

Second-order pooling enhances sensitivity to graph properties.

No single pooling method excels across all evaluated properties.

Abstract

Advancing trustworthy AI requires principled software engineering approaches to model evaluation. Graph Neural Networks (GNNs) have achieved remarkable success in processing graph-structured data, however, their expressiveness in capturing fundamental graph properties remains an open challenge. We address this by developing a property-driven evaluation methodology grounded in formal specification, systematic evaluation, and empirical study. Leveraging Alloy, a software specification language and analyzer, we introduce a configurable graph dataset generator that produces two dataset families: GraphRandom, containing diverse graphs that either satisfy or violate specific properties, and GraphPerturb, introducing controlled structural variations. Together, these benchmarks encompass 336 new datasets, each with at least 10,000 labeled graphs, covering 16 fundamental graph properties critical to distributed systems, knowledge graphs, and biological networks. We propose a general evaluation framework that assesses three key aspects of GNN expressiveness: generalizability, sensitivity, and robustness, with two novel quantitative metrics. Using this framework, we conduct the first comprehensive study on global pooling methods' impact on GNN expressiveness. Our findings reveal distinct trade-offs: attention-based pooling excels in generalization and robustness, while second-order pooling provides superior sensitivity, but no single approach consistently performs well across all properties. These insights highlight fundamental limitations and open research directions including adaptive property-aware pooling, scale-sensitive architectures, and robustness-oriented training. By embedding software engineering rigor into AI evaluation, this work establishes a principled foundation for developing expressive and reliable GNN architectures.