Generate-on-Graph: Treat LLM as both Agent and KG in Incomplete Knowledge Graph Question Answering

TL;DR

This paper introduces Generate-on-Graph (GoG), a training-free method that leverages LLMs as both agents and knowledge sources to improve question answering with incomplete knowledge graphs.

Contribution

The paper proposes a novel training-free approach, GoG, that enables LLMs to generate missing knowledge triples and better handle incomplete KGs in question answering.

Findings

GoG outperforms previous methods on two datasets.

GoG effectively generates missing knowledge triples.

The approach demonstrates improved reasoning over incomplete KGs.

Abstract

To address the issues of insufficient knowledge and hallucination in Large Language Models (LLMs), numerous studies have explored integrating LLMs with Knowledge Graphs (KGs). However, these methods are typically evaluated on conventional Knowledge Graph Question Answering (KGQA) with complete KGs, where all factual triples required for each question are entirely covered by the given KG. In such cases, LLMs primarily act as an agent to find answer entities within the KG, rather than effectively integrating the internal knowledge of LLMs and external knowledge sources such as KGs. In fact, KGs are often incomplete to cover all the knowledge required to answer questions. To simulate these real-world scenarios and evaluate the ability of LLMs to integrate internal and external knowledge, we propose leveraging LLMs for QA under Incomplete Knowledge Graph (IKGQA), where the provided KG lacks some of the factual triples for each question, and construct corresponding datasets. To handle IKGQA, we propose a training-free method called Generate-on-Graph (GoG), which can generate new factual triples while exploring KGs. Specifically, GoG performs reasoning through a Thinking-Searching-Generating framework, which treats LLM as both Agent and KG in IKGQA. Experimental results on two datasets demonstrate that our GoG outperforms all previous methods.