KBM: Delineating Knowledge Boundary for Adaptive Retrieval in Large Language Models

TL;DR

This paper introduces KBM, a model that identifies when retrieval-augmented generation is necessary for large language models, reducing unnecessary retrievals and improving efficiency across various knowledge scenarios.

Contribution

We propose the Knowledge Boundary Model (KBM) to accurately determine when retrieval is needed, enhancing LLM performance and efficiency in dynamic and static knowledge contexts.

Findings

KBM effectively delineates knowledge boundaries in diverse datasets.

Using KBM reduces retrievals while maintaining performance.

KBM performs well as an external plug-in for LLMs.

Abstract

Large Language Models (LLMs) often struggle with dynamically changing knowledge and handling unknown static information. Retrieval-Augmented Generation (RAG) is employed to tackle these challenges and has a significant impact on improving LLM performance. In fact, we find that not all questions need to trigger RAG. By retrieving parts of knowledge unknown to the LLM and allowing the LLM to answer the rest, we can effectively reduce both time and computational costs. In our work, we propose a Knowledge Boundary Model (KBM) to express the known/unknown of a given question, and to determine whether a RAG needs to be triggered. Experiments conducted on 11 English and Chinese datasets illustrate that the KBM effectively delineates the knowledge boundary, significantly decreasing the proportion of retrievals required for optimal end-to-end performance. Furthermore, we evaluate the effectiveness of KBM in three complex scenarios: dynamic knowledge, long-tail static knowledge, and multi-hop problems, as well as its functionality as an external LLM plug-in.