OrthoReg: Improving Graph-regularized MLPs via Orthogonality Regularization

TL;DR

OrthoReg introduces an orthogonality regularization technique to enhance graph-regularized MLPs, addressing their limited expressive power caused by dimensional collapse, and demonstrates improved performance on node classification tasks.

Contribution

The paper identifies dimensional collapse as a key issue in GR-MLPs and proposes OrthoReg, a novel regularization method to improve their expressiveness and performance.

Findings

OrthoReg effectively mitigates dimensional collapse in GR-MLPs.

Experimental results show improved accuracy in semi-supervised and inductive node classification.

OrthoReg outperforms existing GR-MLP models on benchmark datasets.

Abstract

Graph Neural Networks (GNNs) are currently dominating in modeling graph-structure data, while their high reliance on graph structure for inference significantly impedes them from widespread applications. By contrast, Graph-regularized MLPs (GR-MLPs) implicitly inject the graph structure information into model weights, while their performance can hardly match that of GNNs in most tasks. This motivates us to study the causes of the limited performance of GR-MLPs. In this paper, we first demonstrate that node embeddings learned from conventional GR-MLPs suffer from dimensional collapse, a phenomenon in which the largest a few eigenvalues dominate the embedding space, through empirical observations and theoretical analysis. As a result, the expressive power of the learned node representations is constrained. We further propose OrthoReg, a novel GR-MLP model to mitigate the dimensional collapse issue. Through a soft regularization loss on the correlation matrix of node embeddings, OrthoReg explicitly encourages orthogonal node representations and thus can naturally avoid dimensionally collapsed representations. Experiments on traditional transductive semi-supervised classification tasks and inductive node classification for cold-start scenarios demonstrate its effectiveness and superiority.