ROD: Reception-aware Online Distillation for Sparse Graphs

TL;DR

ROD introduces a reception-aware online distillation method that enhances learning on sparse graphs by leveraging multi-scale reception knowledge and peer-teaching, significantly improving performance across multiple graph tasks.

Contribution

The paper presents a novel reception-aware online knowledge distillation framework specifically designed for sparse graph learning, incorporating multi-scale reception signals and dynamic teacher models.

Findings

Achieves state-of-the-art results on 9 datasets.

Improves robustness in sparse graph scenarios.

Enhances performance across node classification, link prediction, and clustering.

Abstract

Graph neural networks (GNNs) have been widely used in many graph-based tasks such as node classification, link prediction, and node clustering. However, GNNs gain their performance benefits mainly from performing the feature propagation and smoothing across the edges of the graph, thus requiring sufficient connectivity and label information for effective propagation. Unfortunately, many real-world networks are sparse in terms of both edges and labels, leading to sub-optimal performance of GNNs. Recent interest in this sparse problem has focused on the self-training approach, which expands supervised signals with pseudo labels. Nevertheless, the self-training approach inherently cannot realize the full potential of refining the learning performance on sparse graphs due to the unsatisfactory quality and quantity of pseudo labels. In this paper, we propose ROD, a novel reception-aware online knowledge distillation approach for sparse graph learning. We design three supervision signals for ROD: multi-scale reception-aware graph knowledge, task-based supervision, and rich distilled knowledge, allowing online knowledge transfer in a peer-teaching manner. To extract knowledge concealed in the multi-scale reception fields, ROD explicitly requires individual student models to preserve different levels of locality information. For a given task, each student would predict based on its reception-scale knowledge, while simultaneously a strong teacher is established on-the-fly by combining multi-scale knowledge. Our approach has been extensively evaluated on 9 datasets and a variety of graph-based tasks, including node classification, link prediction, and node clustering. The result demonstrates that ROD achieves state-of-art performance and is more robust for the graph sparsity.