Implicit Regularization of Mini-Batch Training in Graph Neural Networks

TL;DR

This paper reveals that simple random node sampling in GNN training acts as an implicit regularizer, often outperforming more complex structure-aware methods by reducing gradient variance and aligning mini-batch loss with full-graph loss.

Contribution

The study demonstrates that random node sampling implicitly regularizes GNN training, providing a theoretically grounded, scalable alternative to structure-aware sampling methods.

Findings

RNS matches or outperforms full-graph training on 8 of 10 datasets.

RNS produces mini-batches with lower gradient variance.

Implicit regularization explains the effectiveness of RNS in GNN training.

Abstract

Mini-batch training of Graph Neural Networks (GNNs) is fundamentally different from training on i.i.d. data: sampling a subgraph alters the topology and introduces boundary effects, leading prior work to develop structure-aware samplers that preserve local connectivity and reduce embedding variance. Surprisingly, we demonstrate that the simplest possible scheme, Random Node Sampling (RNS), training on the induced subgraph of uniformly sampled nodes, matches or outperforms full-graph training on 8 of 10 datasets at a fraction of the wall-clock time and memory. To explain this, we apply backward error analysis to graph mini-batch Stochastic Gradient Descent (SGD) and show that it implicitly minimizes the sampled loss plus a regularizer proportional to the mini-batch gradient variance, a quantity directly shaped by the sampler. Although RNS discards local structure, it produces mini-batches whose expected loss is closer to the full-graph loss, and whose per-batch gradients have lower variance, yielding a better implicit objective. Our analysis reframes the choice of graph sampler as a form of implicit regularization, and identifies RNS as a strong, theoretically grounded method for scalable GNN training.