On the Limit of Explaining Black-box Temporal Graph Neural Networks

TL;DR

This paper investigates the limitations of perturbation-based explanation methods for black-box Temporal Graph Neural Networks, revealing that such methods often fail to reliably identify key components and pathways involved in the model's predictions.

Contribution

It demonstrates fundamental limitations of perturbation-based explanations for TGNNs through constructed examples, highlighting challenges in interpretability without internal model access.

Findings

Node-perturbation cannot reliably identify prediction paths.

Edge-perturbation fails to find all contributing nodes.

Perturbing nodes and edges does not reveal temporal aggregation components.

Abstract

Temporal Graph Neural Network (TGNN) has been receiving a lot of attention recently due to its capability in modeling time-evolving graph-related tasks. Similar to Graph Neural Networks, it is also non-trivial to interpret predictions made by a TGNN due to its black-box nature. A major approach tackling this problems in GNNs is by analyzing the model' responses on some perturbations of the model's inputs, called perturbation-based explanation methods. While these methods are convenient and flexible since they do not need internal access to the model, does this lack of internal access prevent them from revealing some important information of the predictions? Motivated by that question, this work studies the limit of some classes of perturbation-based explanation methods. Particularly, by constructing some specific instances of TGNNs, we show (i) node-perturbation cannot reliably identify the paths carrying out the prediction, (ii) edge-perturbation is not reliable in determining all nodes contributing to the prediction and (iii) perturbing both nodes and edges does not reliably help us identify the graph's components carrying out the temporal aggregation in TGNNs.