Knowledge Solver: Teaching LLMs to Search for Domain Knowledge from Knowledge Graphs

TL;DR

This paper introduces Knowledge Solver (KSL), a method that enables large language models to search for domain knowledge from knowledge graphs in a zero-shot manner, improving their reasoning and explainability without retraining external modules.

Contribution

KSL teaches LLMs to perform multi-hop knowledge retrieval from knowledge bases using prompts, eliminating the need for retraining modules and enhancing explainability.

Findings

Significant performance improvements on three datasets.

Enhanced explainability through complete retrieval paths.

Effective zero-shot knowledge retrieval in domain-specific tasks.

Abstract

Large language models (LLMs), such as ChatGPT and GPT-4, are versatile and can solve different tasks due to their emergent ability and generalizability. However, LLMs sometimes lack domain-specific knowledge to perform tasks, which would also cause hallucination during inference. In some previous works, additional modules like graph neural networks (GNNs) are trained on retrieved knowledge from external knowledge bases, aiming to mitigate the problem of lacking domain-specific knowledge. However, incorporating additional modules: 1) would need retraining additional modules when encountering novel domains; 2) would become a bottleneck since LLMs' strong abilities are not fully utilized for retrieval. In this paper, we propose a paradigm, termed Knowledge Solver (KSL), to teach LLMs to search for essential knowledge from external knowledge bases by harnessing their own strong generalizability. Specifically, we design a simple yet effective prompt to transform retrieval into a multi-hop decision sequence, which empowers LLMs with searching knowledge ability in zero-shot manner. Additionally, KSL is able to provide complete retrieval paths and therefore increase explainability of LLMs' reasoning processes. We conduct experiments on three datasets: CommonsenseQA, OpenbookQA, and MedQA-USMLE, and found that our approach improves LLM baseline performance by a relatively large margin.