Self-Knowledge Guided Retrieval Augmentation for Large Language Models

TL;DR

This paper introduces SKR, a method that enables large language models to recognize their knowledge gaps and selectively retrieve external information, improving performance on question-answering tasks.

Contribution

The paper proposes Self-Knowledge guided Retrieval augmentation (SKR), a novel approach that combines internal self-awareness with external retrieval to enhance LLMs' effectiveness.

Findings

SKR outperforms chain-of-thought and fully retrieval-based methods.

SKR improves question-answering accuracy on multiple datasets.

Models using SKR better recognize their knowledge limitations.

Abstract

Large language models (LLMs) have shown superior performance without task-specific fine-tuning. Despite the success, the knowledge stored in the parameters of LLMs could still be incomplete and difficult to update due to the computational costs. As complementary, retrieval-based methods can offer non-parametric world knowledge and improve the performance on tasks such as question answering. However, we find that the retrieved knowledge does not always help and even has a negative impact on original responses occasionally. To better make use of both internal knowledge and external world knowledge, we investigate eliciting the model's ability to recognize what they know and do not know (which is also called self-knowledge) and propose Self-Knowledge guided Retrieval augmentation (SKR), a simple yet effective method which can let LLMs refer to the questions they have previously encountered and adaptively call for external resources when dealing with new questions. We evaluate SKR on multiple datasets and demonstrate that it outperforms chain-of-thought based and fully retrieval-based methods by using either InstructGPT or ChatGPT.