Unveiling Knowledge Utilization Mechanisms in LLM-based Retrieval-Augmented Generation

TL;DR

This paper systematically investigates how large language models (LLMs) utilize internal and external knowledge in retrieval-augmented generation, revealing four key stages and introducing a new neuron identification method to enhance interpretability.

Contribution

It introduces a knowledge stream analysis framework, decomposes the knowledge utilization process into four stages, and proposes KAPE for neuron identification, advancing understanding of LLM knowledge integration.

Findings

Knowledge streaming occurs in four stages: refinement, elicitation, expression, and contestation.

Passage relevance influences the knowledge streaming process.

Deactivation of specific neurons shifts reliance between internal and external knowledge.

Abstract

Considering the inherent limitations of parametric knowledge in large language models (LLMs), retrieval-augmented generation (RAG) is widely employed to expand their knowledge scope. Since RAG has shown promise in knowledge-intensive tasks like open-domain question answering, its broader application to complex tasks and intelligent assistants has further advanced its utility. Despite this progress, the underlying knowledge utilization mechanisms of LLM-based RAG remain underexplored. In this paper, we present a systematic investigation of the intrinsic mechanisms by which LLMs integrate internal (parametric) and external (retrieved) knowledge in RAG scenarios. Specially, we employ knowledge stream analysis at the macroscopic level, and investigate the function of individual modules at the microscopic level. Drawing on knowledge streaming analyses, we decompose the knowledge utilization process into four distinct stages within LLM layers: knowledge refinement, knowledge elicitation, knowledge expression, and knowledge contestation. We further demonstrate that the relevance of passages guides the streaming of knowledge through these stages. At the module level, we introduce a new method, knowledge activation probability entropy (KAPE) for neuron identification associated with either internal or external knowledge. By selectively deactivating these neurons, we achieve targeted shifts in the LLM's reliance on one knowledge source over the other. Moreover, we discern complementary roles for multi-head attention and multi-layer perceptron layers during knowledge formation. These insights offer a foundation for improving interpretability and reliability in retrieval-augmented LLMs, paving the way for more robust and transparent generative solutions in knowledge-intensive domains.