Connecting Graph Convolutional Networks and Graph-Regularized PCA

TL;DR

This paper establishes a mathematical link between graph convolutional networks (GCNs) and graph-regularized PCA, showing that GCN performance is largely driven by graph-based regularization and proposing a GPCA-based initialization to improve GCN training.

Contribution

It reveals a novel connection between GCNs and GPCA, extends GPCA to supervised settings, and introduces a GPCA-based initialization strategy for GCNs and other GNNs.

Findings

GPCA embeddings match or outperform GCN on semi-supervised node classification.

Extending GPCA with ghost edges aligns with supervised GCN performance.

GPCA-based initialization accelerates GCN convergence and enhances robustness.

Abstract

Graph convolution operator of the GCN model is originally motivated from a localized first-order approximation of spectral graph convolutions. This work stands on a different view; establishing a \textit{mathematical connection between graph convolution and graph-regularized PCA} (GPCA). Based on this connection, GCN architecture, shaped by stacking graph convolution layers, shares a close relationship with stacking GPCA. We empirically demonstrate that the \textit{unsupervised} embeddings by GPCA paired with a 1- or 2-layer MLP achieves similar or even better performance than GCN on semi-supervised node classification tasks across five datasets including Open Graph Benchmark \footnote{\url{https://ogb.stanford.edu/}}. This suggests that the prowess of GCN is driven by graph based regularization. In addition, we extend GPCA to the (semi-)supervised setting and show that it is equivalent to GPCA on a graph extended with "ghost" edges between nodes of the same label. Finally, we capitalize on the discovered relationship to design an effective initialization strategy based on stacking GPCA, enabling GCN to converge faster and achieve robust performance at large number of layers. Notably, the proposed initialization is general-purpose and applies to other GNNs.