Teaching MLPs to Master Heterogeneous Graph-Structured Knowledge for Efficient and Accurate Inference

TL;DR

This paper introduces HG2M and HG2M+ frameworks that distill knowledge from Heterogeneous Graph Neural Networks into MLPs, enabling fast, accurate inference on heterogeneous graph data with minimal structural dependency.

Contribution

The paper proposes novel HG2M and HG2M+ methods for knowledge distillation from HGNNs to MLPs, achieving competitive performance and significant inference speedup.

Findings

HG2Ms outperform vanilla MLPs on six datasets.

HG2Ms achieve up to 379.24× inference speedup over HGNNs.

HG2Ms maintain competitive accuracy despite lacking structural dependencies.

Abstract

Heterogeneous Graph Neural Networks (HGNNs) have achieved promising results in various heterogeneous graph learning tasks, owing to their superiority in capturing the intricate relationships and diverse relational semantics inherent in heterogeneous graph structures. However, the neighborhood-fetching latency incurred by structure dependency in HGNNs makes it challenging to deploy for latency-constrained applications that require fast inference. Inspired by recent GNN-to-MLP knowledge distillation frameworks, we introduce HG2M and HG2M+ to combine both HGNN's superior performance and MLP's efficient inference. HG2M directly trains student MLPs with node features as input and soft labels from teacher HGNNs as targets, and HG2M+ further distills reliable and heterogeneous semantic knowledge into student MLPs through reliable node distillation and reliable meta-path distillation. Experiments conducted on six heterogeneous graph datasets show that despite lacking structural dependencies, HG2Ms can still achieve competitive or even better performance than HGNNs and significantly outperform vanilla MLPs. Moreover, HG2Ms demonstrate a 379.24$\times$ speedup in inference over HGNNs on the large-scale IGB-3M-19 dataset, showcasing their ability for latency-sensitive deployments.