Are We Really Measuring Progress? Transferring Insights from Evaluating Recommender Systems to Temporal Link Prediction

TL;DR

This paper critically examines current evaluation practices in Temporal Link Prediction, highlighting issues that may undermine the reliability of benchmarks and proposing directions for more robust and interpretable assessment methods.

Contribution

It identifies key problems in existing TLP evaluation protocols and discusses potential improvements to enhance benchmark reliability and interpretability.

Findings

Current evaluation metrics are inconsistent and sometimes misleading.

Hard negative sampling can distort true model performance.

Assumptions of equal base probabilities across nodes may be invalid.

Abstract

Recent work has questioned the reliability of graph learning benchmarks, citing concerns around task design, methodological rigor, and data suitability. In this extended abstract, we contribute to this discussion by focusing on evaluation strategies in Temporal Link Prediction (TLP). We observe that current evaluation protocols are often affected by one or more of the following issues: (1) inconsistent sampled metrics, (2) reliance on hard negative sampling often introduced as a means to improve robustness, and (3) metrics that implicitly assume equal base probabilities across source nodes by combining predictions. We support these claims through illustrative examples and connections to longstanding concerns in the recommender systems community. Our ongoing work aims to systematically characterize these problems and explore alternatives that can lead to more robust and interpretable evaluation. We conclude with a discussion of potential directions for improving the reliability of TLP benchmarks.