ZeroKBC: A Comprehensive Benchmark for Zero-Shot Knowledge Base Completion

TL;DR

This paper introduces ZeroKBC, a comprehensive benchmark for zero-shot knowledge base completion, highlighting the challenges and gaps in current systems when dealing with unseen entities and relations.

Contribution

The paper systematically defines zero-shot KBC scenarios, develops a diverse benchmark, and evaluates existing systems, revealing their limitations and guiding future research directions.

Findings

Current KBC systems perform poorly on ZeroKBC.

ZeroKBC covers diverse zero-shot scenarios.

Analysis suggests directions for improving zero-shot KBC.

Abstract

Knowledge base completion (KBC) aims to predict the missing links in knowledge graphs. Previous KBC tasks and approaches mainly focus on the setting where all test entities and relations have appeared in the training set. However, there has been limited research on the zero-shot KBC settings, where we need to deal with unseen entities and relations that emerge in a constantly growing knowledge base. In this work, we systematically examine different possible scenarios of zero-shot KBC and develop a comprehensive benchmark, ZeroKBC, that covers these scenarios with diverse types of knowledge sources. Our systematic analysis reveals several missing yet important zero-shot KBC settings. Experimental results show that canonical and state-of-the-art KBC systems cannot achieve satisfactory performance on this challenging benchmark. By analyzing the strength and weaknesses of these systems on solving ZeroKBC, we further present several important observations and promising future directions.