Prompting Large Language Models with Partial Knowledge for Answering Questions with Unseen Entities

TL;DR

This paper explores how large language models can be prompted with partially relevant knowledge to improve answering questions involving unseen entities, addressing challenges in incomplete knowledge bases.

Contribution

It introduces a novel awakening approach that leverages embedded partially relevant knowledge in LLMs, supported by theoretical analysis and experiments on KGQA datasets.

Findings

Awakening effect enhances LLM performance on unseen entity questions.

The approach outperforms embedding-based similarity methods.

Demonstrates effectiveness in real-world incomplete knowledge scenarios.

Abstract

Retrieval-Augmented Generation (RAG) shows impressive performance by supplementing and substituting parametric knowledge in Large Language Models (LLMs). Retrieved knowledge can be divided into three types: explicit answer evidence, implicit answer clue, and insufficient answer context which can be further categorized into totally irrelevant and partially relevant information. Effectively utilizing partially relevant knowledge remains a key challenge for RAG systems, especially in incomplete knowledge base retrieval. Contrary to the conventional view, we propose a new perspective: LLMs can be awakened via partially relevant knowledge already embedded in LLMs. To comprehensively investigate this phenomenon, the triplets located in the gold reasoning path and their variants are used to construct partially relevant knowledge by removing the path that contains the answer. We provide theoretical analysis of the awakening effect in LLMs and support our hypothesis with experiments on two Knowledge Graphs (KGs) Question Answering (QA) datasets. Furthermore, we present a new task, Unseen Entity KGQA, simulating real-world challenges where entity linking fails due to KG incompleteness. Our awakening-based approach demonstrates greater efficacy in practical applications, outperforms traditional methods that rely on embedding-based similarity which are prone to returning noisy information.