Layer-wise training for self-supervised learning on graphs

TL;DR

This paper introduces a layer-wise training algorithm for GNNs that improves efficiency and mitigates oversmoothing, enabling deeper models on large graphs without high memory costs.

Contribution

It proposes Layer-wise Regularized Graph Infomax, a novel layer-wise training method for GNNs that decouples feature propagation and transformation, enhancing scalability and performance.

Findings

Achieves similar accuracy to end-to-end methods on large graphs

Significantly reduces memory and computational requirements

Prevents oversmoothing in deep GNNs

Abstract

End-to-end training of graph neural networks (GNN) on large graphs presents several memory and computational challenges, and limits the application to shallow architectures as depth exponentially increases the memory and space complexities. In this manuscript, we propose Layer-wise Regularized Graph Infomax, an algorithm to train GNNs layer by layer in a self-supervised manner. We decouple the feature propagation and feature transformation carried out by GNNs to learn node representations in order to derive a loss function based on the prediction of future inputs. We evaluate the algorithm in inductive large graphs and show similar performance to other end to end methods and a substantially increased efficiency, which enables the training of more sophisticated models in one single device. We also show that our algorithm avoids the oversmoothing of the representations, another common challenge of deep GNNs.