Knowledge Localization: Mission Not Accomplished? Enter Query Localization!

TL;DR

This paper critically re-examines the Knowledge Localization assumption in LLMs, introduces the Query Localization concept, and proposes a new method to improve knowledge modification by considering both storage and expression mechanisms.

Contribution

It challenges the traditional Knowledge Localization assumption, introduces the Query Localization framework, and proposes a new knowledge modification method based on this framework.

Findings

KL assumption has limitations in knowledge storage and expression.

Query Localization (QL) provides a more flexible understanding of knowledge localization.

The proposed method improves knowledge modification performance.

Abstract

Large language models (LLMs) store extensive factual knowledge, but the mechanisms behind how they store and express this knowledge remain unclear. The Knowledge Neuron (KN) thesis is a prominent theory for explaining these mechanisms. This theory is based on the Knowledge Localization (KL) assumption, which suggests that a fact can be localized to a few knowledge storage units, namely knowledge neurons. However, this assumption has two limitations: first, it may be too rigid regarding knowledge storage, and second, it neglects the role of the attention module in knowledge expression. In this paper, we first re-examine the KL assumption and demonstrate that its limitations do indeed exist. To address these, we then present two new findings, each targeting one of the limitations: one focusing on knowledge storage and the other on knowledge expression. We summarize these findings as \textbf{Query Localization} (QL) assumption and argue that the KL assumption can be viewed as a simplification of the QL assumption. Based on QL assumption, we further propose the Consistency-Aware KN modification method, which improves the performance of knowledge modification, further validating our new assumption. We conduct 39 sets of experiments, along with additional visualization experiments, to rigorously confirm our conclusions. Code is available at https://github.com/heng840/KnowledgeLocalization.