Does Knowledge Localization Hold True? Surprising Differences Between Entity and Relation Perspectives in Language Models

TL;DR

This paper explores how language models store entity and relational knowledge, revealing they are stored differently and cannot be directly transferred, with relational knowledge also encoded in attention modules.

Contribution

It uncovers the distinct storage mechanisms for entity and relational knowledge in language models and challenges prior assumptions about knowledge localization.

Findings

Entity and relational knowledge are not directly transferable.

Relational knowledge is stored in both MLP weights and attention modules.

Knowledge editing impacts entity and relation knowledge differently.

Abstract

Large language models encapsulate knowledge and have demonstrated superior performance on various natural language processing tasks. Recent studies have localized this knowledge to specific model parameters, such as the MLP weights in intermediate layers. This study investigates the differences between entity and relational knowledge through knowledge editing. Our findings reveal that entity and relational knowledge cannot be directly transferred or mapped to each other. This result is unexpected, as logically, modifying the entity or the relation within the same knowledge triplet should yield equivalent outcomes. To further elucidate the differences between entity and relational knowledge, we employ causal analysis to investigate how relational knowledge is stored in pre-trained models. Contrary to prior research suggesting that knowledge is stored in MLP weights, our experiments demonstrate that relational knowledge is also significantly encoded in attention modules. This insight highlights the multifaceted nature of knowledge storage in language models, underscoring the complexity of manipulating specific types of knowledge within these models.