Temporal graph models fail to capture global temporal dynamics

TL;DR

This paper critically evaluates temporal graph models, revealing their failure to capture global dynamics, and introduces simple yet effective baselines and measures to better understand and improve modeling of datasets with strong temporal changes.

Contribution

It presents a trivial baseline outperforming existing methods, introduces Wasserstein-based measures for global dynamics, and proposes improved negative sampling schemes for temporal graph modeling.

Findings

Simple baseline outperforms complex models on large datasets

Negative sampling can cause model degeneration during training

Proposed measures quantify global temporal dynamics effectively

Abstract

A recently released Temporal Graph Benchmark is analyzed in the context of Dynamic Link Property Prediction. We outline our observations and propose a trivial optimization-free baseline of "recently popular nodes" outperforming other methods on medium and large-size datasets in the Temporal Graph Benchmark. We propose two measures based on Wasserstein distance which can quantify the strength of short-term and long-term global dynamics of datasets. By analyzing our unexpectedly strong baseline, we show how standard negative sampling evaluation can be unsuitable for datasets with strong temporal dynamics. We also show how simple negative-sampling can lead to model degeneration during training, resulting in impossible to rank, fully saturated predictions of temporal graph networks. We propose improved negative sampling schemes for both training and evaluation and prove their usefulness. We conduct a comparison with a model trained non-contrastively without negative sampling. Our results provide a challenging baseline and indicate that temporal graph network architectures need deep rethinking for usage in problems with significant global dynamics, such as social media, cryptocurrency markets or e-commerce. We open-source the code for baselines, measures and proposed negative sampling schemes.