Understanding the Mechanisms Behind Structural Influences on Link Prediction: A Case Study on FB15k-237

TL;DR

This paper investigates how structural features of the FB15k-237 dataset influence link prediction performance, revealing that relationship category distribution and subgraph structure significantly impact model effectiveness and offering insights for future dataset design.

Contribution

It introduces a structured subgraph sampling method and uses correlation, sensitivity, and LIME analyses to uncover the mechanisms by which dataset structure affects link prediction performance.

Findings

Relationship category distribution significantly affects performance

Size of strongly connected components influences results

Relationship categories modulate importance of embeddings

Abstract

FB15k-237 mitigates the data leakage issue by excluding inverse and symmetric relationship triples, however, this has led to substantial performance degradation and slow improvement progress. Traditional approaches demonstrate limited effectiveness on FB15k-237, primarily because the underlying mechanism by which structural features of the dataset influence model performance remains unexplored. To bridge this gap, we systematically investigate the impact mechanism of dataset structural features on link prediction performance. Firstly, we design a structured subgraph sampling strategy that ensures connectivity while constructing subgraphs with distinct structural features. Then, through correlation and sensitivity analyses conducted across several mainstream models, we observe that the distribution of relationship categories within subgraphs significantly affects performance, followed by the size of strongly connected components. Further exploration using the LIME model clarifies the intrinsic mechanism by which relationship categories influence link prediction performance, revealing that relationship categories primarily modulate the relative importance between entity embeddings and relationship embeddings and relationship embeddings, thereby affecting link prediction outcomes. These findings provide theoretical insights for addressing performance bottlenecks on FB15k-237, while the proposed analytical framework also offers methodological guidance for future studies dealing with structurally constrained datasets.