Retrieval Heads are Dynamic

TL;DR

This paper reveals that retrieval heads in Large Language Models are dynamically changing during generation, with implications for understanding model internal mechanisms and improving retrieval-augmented tasks.

Contribution

It introduces the concept of dynamic retrieval heads, demonstrating their variability, irreplaceability, and internal planning role in LLMs, supported by extensive analysis and experiments.

Findings

Retrieval heads vary across timesteps during generation.

Dynamic retrieval heads are specific and cannot be replaced by static ones.

Hidden states encode signals predicting future retrieval head patterns.

Abstract

Recent studies have identified "retrieval heads" in Large Language Models (LLMs) responsible for extracting information from input contexts. However, prior works largely rely on static statistics aggregated across datasets, identifying heads that perform retrieval on average. This perspective overlooks the fine-grained temporal dynamics of autoregressive generation. In this paper, we investigate retrieval heads from a dynamic perspective. Through extensive analysis, we establish three core claims: (1) Dynamism: Retrieval heads vary dynamically across timesteps; (2) Irreplaceability: Dynamic retrieval heads are specific at each timestep and cannot be effectively replaced by static retrieval heads; and (3) Correlation: The model's hidden state encodes a predictive signal for future retrieval head patterns, indicating an internal planning mechanism. We validate these findings on the Needle-in-a-Haystack task and a multi-hop QA task, and quantify the differences on the utility of dynamic and static retrieval heads in a Dynamic Retrieval-Augmented Generation framework. Our study provides new insights into the internal mechanisms of LLMs.