Alignahead: Online Cross-Layer Knowledge Extraction on Graph Neural Networks

TL;DR

This paper introduces an online cross-layer knowledge distillation method for GNNs that improves student model performance without pre-trained teachers, accelerates convergence, and enhances knowledge sharing across layers.

Contribution

Proposes a novel online distillation framework with cross-layer alignment for GNNs, eliminating the need for pre-trained teacher models and improving training efficiency.

Findings

Consistent performance boost across five datasets.

Accelerated convergence speed with the alignahead technique.

Effectiveness increases with more student models in training.

Abstract

Existing knowledge distillation methods on graph neural networks (GNNs) are almost offline, where the student model extracts knowledge from a powerful teacher model to improve its performance. However, a pre-trained teacher model is not always accessible due to training cost, privacy, etc. In this paper, we propose a novel online knowledge distillation framework to resolve this problem. Specifically, each student GNN model learns the extracted local structure from another simultaneously trained counterpart in an alternating training procedure. We further develop a cross-layer distillation strategy by aligning ahead one student layer with the layer in different depth of another student model, which theoretically makes the structure information spread over all layers. Experimental results on five datasets including PPI, Coauthor-CS/Physics and Amazon-Computer/Photo demonstrate that the student performance is consistently boosted in our collaborative training framework without the supervision of a pre-trained teacher model. In addition, we also find that our alignahead technique can accelerate the model convergence speed and its effectiveness can be generally improved by increasing the student numbers in training. Code is available: https://github.com/GuoJY-eatsTG/Alignahead