Quantifying discriminability of evaluation metrics in link prediction for real networks

TL;DR

This paper introduces a measure to quantify how well different evaluation metrics distinguish between link prediction algorithms across various real networks, highlighting the superior discriminability of H-measure and AUC.

Contribution

It proposes a novel discriminability measure for evaluation metrics and systematically compares eight metrics across 131 real networks and 20 algorithms.

Findings

H-measure and AUC have the strongest discriminability.

NDCG also shows high discriminability.

Discriminability results are consistent across different network domains.

Abstract

Link prediction is one of the most productive branches in network science, aiming to predict links that would have existed but have not yet been observed, or links that will appear during the evolution of the network. Over nearly two decades, the field of link prediction has amassed a substantial body of research, encompassing a plethora of algorithms and diverse applications. For any algorithm, one or more evaluation metrics are required to assess its performance. Because using different evaluation metrics can provide different assessments of the algorithm performance, how to select appropriate evaluation metrics is a fundamental issue in link prediction. To address this issue, we propose a novel measure that quantifiers the discriminability of any evaluation metric given a real network and an algorithm. Based on 131 real networks and 20 representative algorithms, we systematically compare the discriminabilities of eight evaluation metrics, and demonstrate that H-measure and Area Under the ROC Curve (AUC) exhibit the strongest discriminabilities, followed by Normalized Discounted Cumulative Gain (NDCG). Our finding is robust for networks in different domains and algorithms of different types. This study provides insights into the selection of evaluation metrics, which may further contribute to standardizing the evaluating process of link prediction algorithms.