On the Prediction Instability of Graph Neural Networks

TL;DR

This paper investigates the prediction instability of Graph Neural Networks, revealing that different training runs can produce significantly different node-level predictions despite similar overall performance, impacting reliability.

Contribution

It systematically assesses GNN prediction instability, highlighting factors influencing stability and showing that higher performance often correlates with reduced instability.

Findings

Multiple GNN training runs yield similar overall accuracy but different node predictions.

Up to one third of misclassified nodes differ across runs.

Increasing training data and tuning hyperparameters can improve prediction stability.

Abstract

Instability of trained models, i.e., the dependence of individual node predictions on random factors, can affect reproducibility, reliability, and trust in machine learning systems. In this paper, we systematically assess the prediction instability of node classification with state-of-the-art Graph Neural Networks (GNNs). With our experiments, we establish that multiple instantiations of popular GNN models trained on the same data with the same model hyperparameters result in almost identical aggregated performance but display substantial disagreement in the predictions for individual nodes. We find that up to one third of the incorrectly classified nodes differ across algorithm runs. We identify correlations between hyperparameters, node properties, and the size of the training set with the stability of predictions. In general, maximizing model performance implicitly also reduces model instability.