Graph Neural Networks Use Graphs When They Shouldn't

TL;DR

This paper investigates how Graph Neural Networks tend to overfit to input graph structures even when ignoring the graph would yield better results, analyzing the implicit bias of gradient descent and proposing solutions using regular graphs.

Contribution

It provides a theoretical analysis of GNN overfitting to graph structures, especially on regular graphs, and demonstrates practical methods to mitigate this issue.

Findings

GNNs tend to overfit to the input graph structure even when ignoring it is optimal.

Regular graphs are more robust to GNN overfitting and guarantee better extrapolation.

Using regular graphs can improve GNN performance and reduce overfitting in real-world applications.

Abstract

Predictions over graphs play a crucial role in various domains, including social networks and medicine. Graph Neural Networks (GNNs) have emerged as the dominant approach for learning on graph data. Although a graph-structure is provided as input to the GNN, in some cases the best solution can be obtained by ignoring it. While GNNs have the ability to ignore the graph- structure in such cases, it is not clear that they will. In this work, we show that GNNs actually tend to overfit the given graph-structure. Namely, they use it even when a better solution can be obtained by ignoring it. We analyze the implicit bias of gradient-descent learning of GNNs and prove that when the ground truth function does not use the graphs, GNNs are not guaranteed to learn a solution that ignores the graph, even with infinite data. We examine this phenomenon with respect to different graph distributions and find that regular graphs are more robust to this over-fitting. We also prove that within the family of regular graphs, GNNs are guaranteed to extrapolate when learning with gradient descent. Finally, based on our empirical and theoretical findings, we demonstrate on real-data how regular graphs can be leveraged to reduce graph overfitting and enhance performance.