Quantifying the Optimization and Generalization Advantages of Graph Neural Networks Over Multilayer Perceptrons

TL;DR

This paper provides a quantitative analysis showing that graph neural networks (GNNs) outperform multilayer perceptrons (MLPs) in optimization and generalization by emphasizing signal learning, especially in low test error regimes, supported by theoretical and empirical evidence.

Contribution

It offers a novel theoretical comparison of GNNs and MLPs from an optimization and generalization perspective using a signal-noise model.

Findings

GNNs prioritize signal learning over noise more than MLPs.

GNNs achieve significantly lower test error regimes, proportional to D^{q-2}.

Empirical results on synthetic and real datasets support the theoretical analysis.

Abstract

Graph neural networks (GNNs) have demonstrated remarkable capabilities in learning from graph-structured data, often outperforming traditional Multilayer Perceptrons (MLPs) in numerous graph-based tasks. Although existing works have demonstrated the benefits of graph convolution through Laplacian smoothing, expressivity or separability, there remains a lack of quantitative analysis comparing GNNs and MLPs from an optimization and generalization perspective. This study aims to address this gap by examining the role of graph convolution through feature learning theory. Using a signal-noise data model, we conduct a comparative analysis of the optimization and generalization between two-layer graph convolutional networks (GCNs) and their MLP counterparts. Our approach tracks the trajectory of signal learning and noise memorization in GNNs, characterizing their post-training generalization. We reveal that GNNs significantly prioritize signal learning, thus enhancing the regime of {low test error} over MLPs by $D^{q-2}$ times, where $D$ denotes a node's expected degree and $q$ is the power of ReLU activation function with $q>2$. This finding highlights a substantial and quantitative discrepancy between GNNs and MLPs in terms of optimization and generalization, a conclusion further supported by our empirical simulations on both synthetic and real-world datasets.