Online Cross-Layer Knowledge Distillation on Graph Neural Networks with Deep Supervision

TL;DR

This paper introduces Alignahead++, an online cross-layer knowledge distillation method for GNNs that employs deep supervision and collaborative training to improve student model performance without pre-trained teachers.

Contribution

Proposes a novel online distillation framework with deep supervision for GNNs, enabling effective training without pre-trained teacher models and enhancing performance through multi-student collaboration.

Findings

Consistently improves student GNN performance across datasets

Effectiveness increases with more student models

Avoids over-smoothing with deep supervision

Abstract

Graph neural networks (GNNs) have become one of the most popular research topics in both academia and industry communities for their strong ability in handling irregular graph data. However, large-scale datasets are posing great challenges for deploying GNNs in edge devices with limited resources and model compression techniques have drawn considerable research attention. Existing model compression techniques such as knowledge distillation (KD) mainly focus on convolutional neural networks (CNNs). Only limited attempts have been made recently for distilling knowledge from GNNs in an offline manner. As the performance of the teacher model does not necessarily improve as the number of layers increases in GNNs, selecting an appropriate teacher model will require substantial efforts. To address these challenges, we propose a novel online knowledge distillation framework called Alignahead++ in this paper. Alignahead++ transfers structure and feature information in a student layer to the previous layer of another simultaneously trained student model in an alternating training procedure. Meanwhile, to avoid over-smoothing problem in GNNs, deep supervision is employed in Alignahead++ by adding an auxiliary classifier in each intermediate layer to prevent the collapse of the node feature embeddings. Experimental results on four datasets including PPI, Cora, PubMed and CiteSeer demonstrate that the student performance is consistently boosted in our collaborative training framework without the supervision of a pre-trained teacher model and its effectiveness can generally be improved by increasing the number of students.