InfGraND: An Influence-Guided GNN-to-MLP Knowledge Distillation

TL;DR

InfGraND introduces an influence-guided knowledge distillation framework that enhances MLPs' performance on graph tasks by focusing on structurally influential nodes and pre-computing neighborhood features, improving efficiency and accuracy.

Contribution

The paper proposes a novel influence-guided graph knowledge distillation method that prioritizes structurally important nodes and incorporates multi-hop neighborhood features for improved GNN-to-MLP transfer.

Findings

Outperforms prior GNN-to-MLP KD methods on seven benchmark datasets.

Effective in both transductive and inductive settings.

Enables low-latency inference with high accuracy.

Abstract

Graph Neural Networks (GNNs) are the go-to model for graph data analysis. However, GNNs rely on two key operations - aggregation and update, which can pose challenges for low-latency inference tasks or resource-constrained scenarios. Simple Multi-Layer Perceptrons (MLPs) offer a computationally efficient alternative. Yet, training an MLP in a supervised setting often leads to suboptimal performance. Knowledge Distillation (KD) from a GNN teacher to an MLP student has emerged to bridge this gap. However, most KD methods either transfer knowledge uniformly across all nodes or rely on graph-agnostic indicators such as prediction uncertainty. We argue this overlooks a more fundamental, graph-centric inquiry: "How important is a node to the structure of the graph?" We introduce a framework, InfGraND, an Influence-guided Graph KNowledge Distillation from GNN to MLP that addresses this by identifying and prioritizing structurally influential nodes to guide the distillation process, ensuring that the MLP learns from the most critical parts of the graph. Additionally, InfGraND embeds structural awareness in MLPs through one-time multi-hop neighborhood feature pre-computation, which enriches the student MLP's input and thus avoids inference-time overhead. Our rigorous evaluation in transductive and inductive settings across seven homophilic graph benchmark datasets shows InfGraND consistently outperforms prior GNN to MLP KD methods, demonstrating its practicality for numerous latency-critical applications in real-world settings.