How Graph Neural Networks Learn: Lessons from Training Dynamics

TL;DR

This paper investigates the training dynamics of graph neural networks, revealing how they implicitly leverage graph structure during learning, and proposes a simple, fast algorithm that rivals GNN performance.

Contribution

It introduces the concept of kernel-graph alignment, provides theoretical insights into GNN learning, and proposes a parameter-free, efficient update method.

Findings

Kernel-graph alignment explains GNN learning behavior.

The proposed sparse matrix update matches GNN accuracy.

The method is significantly faster than traditional GNN training.

Abstract

A long-standing goal in deep learning has been to characterize the learning behavior of black-box models in a more interpretable manner. For graph neural networks (GNNs), considerable advances have been made in formalizing what functions they can represent, but whether GNNs will learn desired functions during the optimization process remains less clear. To fill this gap, we study their training dynamics in function space. In particular, we find that the gradient descent optimization of GNNs implicitly leverages the graph structure to update the learned function, as can be quantified by a phenomenon which we call \emph{kernel-graph alignment}. We provide theoretical explanations for the emergence of this phenomenon in the overparameterized regime and empirically validate it on real-world GNNs. This finding offers new interpretable insights into when and why the learned GNN functions generalize, highlighting their limitations in heterophilic graphs. Practically, we propose a parameter-free algorithm that directly uses a sparse matrix (i.e. graph adjacency) to update the learned function. We demonstrate that this embarrassingly simple approach can be as effective as GNNs while being orders-of-magnitude faster.