Comparing discriminating abilities of evaluation metrics in link prediction

TL;DR

This paper evaluates the discriminating abilities of various link prediction evaluation metrics using an artificial network model, finding AUC, AUPR, and NDCG outperform others in distinguishing prediction accuracy.

Contribution

It introduces a novel framework and artificial network model to quantitatively compare the discriminating abilities of nine link prediction evaluation metrics.

Findings

AUC, AUPR, and NDCG have higher discriminating abilities.

The framework effectively assesses metric effectiveness.

The artificial network model allows controlled variation of prediction accuracy.

Abstract

Link prediction aims to predict the potential existence of links between two unconnected nodes within a network based on the known topological characteristics. Evaluation metrics are used to assess the effectiveness of algorithms in link prediction. The discriminating ability of these evaluation metrics is vitally important for accurately evaluating link prediction algorithms. In this study, we propose an artificial network model, based on which one can adjust a single parameter to monotonically and continuously turn the prediction accuracy of the specifically designed link prediction algorithm. Building upon this foundation, we show a framework to depict the effectiveness of evaluating metrics by focusing on their discriminating ability. Specifically, a quantitative comparison in the abilities of correctly discerning varying prediction accuracies was conducted encompassing nine evaluation metrics: Precision, Recall, F1-Measure, Matthews Correlation Coefficient (MCC), Balanced Precision (BP), the Area Under the receiver operating characteristic Curve (AUC), the Area Under the Precision-Recall curve (AUPR), Normalized Discounted Cumulative Gain (NDCG), and the Area Under the magnified ROC (AUC-mROC). The results indicate that the discriminating abilities of the three metrics, AUC, AUPR, and NDCG, are significantly higher than those of other metrics.