AutoWeird: Weird Translational Scoring Function Identified by Random Search

TL;DR

AutoWeird is a simple, randomly discovered scoring function for knowledge graph triplet plausibility that performs well on one dataset due to evaluation issues and tail entity distribution, highlighting the importance of proper evaluation.

Contribution

The paper introduces AutoWeird, a novel scoring function found by random search, and analyzes its performance and the impact of dataset characteristics on link prediction evaluation.

Findings

AutoWeird achieves top-1 on ogbl-wikikg2 due to evaluation bias.

AutoWeird performs poorly on ogbl-biokg, indicating dataset dependency.

Evaluation protocols significantly influence perceived method effectiveness.

Abstract

Scoring function (SF) measures the plausibility of triplets in knowledge graphs. Different scoring functions can lead to huge differences in link prediction performances on different knowledge graphs. In this report, we describe a weird scoring function found by random search on the open graph benchmark (OGB). This scoring function, called AutoWeird, only uses tail entity and relation in a triplet to compute its plausibility score. Experimental results show that AutoWeird achieves top-1 performance on ogbl-wikikg2 data set, but has much worse performance than other methods on ogbl-biokg data set. By analyzing the tail entity distribution and evaluation protocol of these two data sets, we attribute the unexpected success of AutoWeird on ogbl-wikikg2 to inappropriate evaluation and concentrated tail entity distribution. Such results may motivate further research on how to accurately evaluate the performance of different link prediction methods for knowledge graphs.