On Evaluating Embedding Models for Knowledge Base Completion

TL;DR

This paper critically examines the effectiveness of current embedding models for knowledge base completion, revealing that they underperform under more appropriate evaluation protocols and highlighting the need for improved models and assessment methods.

Contribution

It challenges existing evaluation practices, demonstrating that current models are inadequate for knowledge base completion when assessed with better-suited metrics.

Findings

Current models perform poorly on revised evaluation tasks.

Simple rule-based baselines outperform many embedding models.

Existing evaluation protocols favor question answering over knowledge base completion.

Abstract

Knowledge bases contribute to many web search and mining tasks, yet they are often incomplete. To add missing facts to a given knowledge base, various embedding models have been proposed in the recent literature. Perhaps surprisingly, relatively simple models with limited expressiveness often performed remarkably well under today's most commonly used evaluation protocols. In this paper, we explore whether recent models work well for knowledge base completion and argue that the current evaluation protocols are more suited for question answering rather than knowledge base completion. We show that when focusing on a different prediction task for evaluating knowledge base completion, the performance of current embedding models is unsatisfactory even on datasets previously thought to be too easy. This is especially true when embedding models are compared against a simple rule-based baseline. This work indicates the need for more research into the embedding models and evaluation protocols for knowledge base completion.