Distilling Influences to Mitigate Prediction Churn in Graph Neural Networks

TL;DR

This paper investigates prediction churn in graph neural networks, introduces the Influence Difference metric, and proposes DropDistillation to improve model stability and performance in knowledge distillation tasks.

Contribution

It introduces the Influence Difference metric and DropDistillation method to reduce prediction churn and enhance stability in graph neural network knowledge distillation.

Findings

DropDistillation outperforms previous methods in stability and performance

Influence Difference effectively quantifies feature utilization differences

Models with minimized influence difference show improved prediction consistency

Abstract

Models with similar performances exhibit significant disagreement in the predictions of individual samples, referred to as prediction churn. Our work explores this phenomenon in graph neural networks by investigating differences between models differing only in their initializations in their utilized features for predictions. We propose a novel metric called Influence Difference (ID) to quantify the variation in reasons used by nodes across models by comparing their influence distribution. Additionally, we consider the differences between nodes with a stable and an unstable prediction, positing that both equally utilize different reasons and thus provide a meaningful gradient signal to closely match two models even when the predictions for nodes are similar. Based on our analysis, we propose to minimize this ID in Knowledge Distillation, a domain where a new model should closely match an established one. As an efficient approximation, we introduce DropDistillation (DD) that matches the output for a graph perturbed by edge deletions. Our empirical evaluation of six benchmark datasets for node classification validates the differences in utilized features. DD outperforms previous methods regarding prediction stability and overall performance in all considered Knowledge Distillation experiments.