Temporal-Aware Evaluation and Learning for Temporal Graph Neural Networks

TL;DR

This paper critically examines evaluation metrics for Temporal Graph Neural Networks, identifies their shortcomings in capturing volatility clustering, and proposes a new volatility-aware metric and training approach to improve temporal performance analysis.

Contribution

It introduces a novel volatility-aware evaluation metric and training objective for TGNNs, addressing the inadequacies of existing metrics in capturing temporal error clustering.

Findings

Existing metrics fail to detect volatility clustering in TGNN errors.

Different TGNN mechanisms exhibit distinct error clustering patterns.

The proposed metric and training objective reduce volatility clustering in errors.

Abstract

Temporal Graph Neural Networks (TGNNs) are a family of graph neural networks designed to model and learn dynamic information from temporal graphs. Given their substantial empirical success, there is an escalating interest in TGNNs within the research community. However, the majority of these efforts have been channelled towards algorithm and system design, with the evaluation metrics receiving comparatively less attention. Effective evaluation metrics are crucial for providing detailed performance insights, particularly in the temporal domain. This paper investigates the commonly used evaluation metrics for TGNNs and illustrates the failure mechanisms of these metrics in capturing essential temporal structures in the predictive behaviour of TGNNs. We provide a mathematical formulation of existing performance metrics and utilize an instance-based study to underscore their inadequacies in identifying volatility clustering (the occurrence of emerging errors within a brief interval). This phenomenon has profound implications for both algorithm and system design in the temporal domain. To address this deficiency, we introduce a new volatility-aware evaluation metric (termed volatility cluster statistics), designed for a more refined analysis of model temporal performance. Additionally, we demonstrate how this metric can serve as a temporal-volatility-aware training objective to alleviate the clustering of temporal errors. Through comprehensive experiments on various TGNN models, we validate our analysis and the proposed approach. The empirical results offer revealing insights: 1) existing TGNNs are prone to making errors with volatility clustering, and 2) TGNNs with different mechanisms to capture temporal information exhibit distinct volatility clustering patterns. Our empirical findings demonstrate that our proposed training objective effectively reduces volatility clusters in error.