Towards Bridging Generalization and Expressivity of Graph Neural Networks

TL;DR

This paper investigates the relationship between expressivity and generalization in graph neural networks, proposing a theoretical framework and bounds that explain how expressive GNNs can still generalize well across diverse graph structures.

Contribution

The paper introduces a novel variance margin-based generalization bound for GNNs that is architecture-agnostic and links structural expressivity to generalization performance.

Findings

Expressive GNNs can generalize effectively despite potential overfitting risks.

A new theoretical framework connects GNN expressivity to variance in graph structures.

Empirical results support the theoretical analysis and highlight the trade-off between intra-class and inter-class properties.

Abstract

Expressivity and generalization are two critical aspects of graph neural networks (GNNs). While significant progress has been made in studying the expressivity of GNNs, much less is known about their generalization capabilities, particularly when dealing with the inherent complexity of graph-structured data. In this work, we address the intricate relationship between expressivity and generalization in GNNs. Theoretical studies conjecture a trade-off between the two: highly expressive models risk overfitting, while those focused on generalization may sacrifice expressivity. However, empirical evidence often contradicts this assumption, with expressive GNNs frequently demonstrating strong generalization. We explore this contradiction by introducing a novel framework that connects GNN generalization to the variance in graph structures they can capture. This leads us to propose a $k$-variance margin-based generalization bound that characterizes the structural properties of graph embeddings in terms of their upper-bounded expressive power. Our analysis does not rely on specific GNN architectures, making it broadly applicable across GNN models. We further uncover a trade-off between intra-class concentration and inter-class separation, both of which are crucial for effective generalization. Through case studies and experiments on real-world datasets, we demonstrate that our theoretical findings align with empirical results, offering a deeper understanding of how expressivity can enhance GNN generalization.