Assessing LLMs Suitability for Knowledge Graph Completion

TL;DR

This paper evaluates the potential of large language models for knowledge graph completion, highlighting their capabilities and limitations, and demonstrating that with proper prompting, they can perform well in specific tasks.

Contribution

It provides an empirical assessment of LLMs on knowledge graph completion tasks, analyzing their performance with different prompts and in various settings.

Findings

LLMs can perform knowledge graph completion with proper prompts

Prompt design significantly impacts LLM performance

LLMs show promise but also exhibit hallucinations and non-determinism

Abstract

Recent work has shown the capability of Large Language Models (LLMs) to solve tasks related to Knowledge Graphs, such as Knowledge Graph Completion, even in Zero- or Few-Shot paradigms. However, they are known to hallucinate answers, or output results in a non-deterministic manner, thus leading to wrongly reasoned responses, even if they satisfy the user's demands. To highlight opportunities and challenges in knowledge graphs-related tasks, we experiment with three distinguished LLMs, namely Mixtral-8x7b-Instruct-v0.1, GPT-3.5-Turbo-0125 and GPT-4o, on Knowledge Graph Completion for static knowledge graphs, using prompts constructed following the TELeR taxonomy, in Zero- and One-Shot contexts, on a Task-Oriented Dialogue system use case. When evaluated using both strict and flexible metrics measurement manners, our results show that LLMs could be fit for such a task if prompts encapsulate sufficient information and relevant examples.