What does the Knowledge Neuron Thesis Have to do with Knowledge?

TL;DR

This paper critically reevaluates the Knowledge Neuron Thesis, arguing that it oversimplifies how large language models represent and recall factual knowledge, and suggests exploring more complex model components for understanding knowledge representation.

Contribution

The paper challenges the KN thesis by demonstrating its limitations and proposing that knowledge is not solely stored in MLP weights, advocating for broader analysis of model structures.

Findings

KN thesis oversimplifies knowledge recall

MLP weights do not fully encode knowledge

Attention mechanisms and layer structures are crucial

Abstract

We reassess the Knowledge Neuron (KN) Thesis: an interpretation of the mechanism underlying the ability of large language models to recall facts from a training corpus. This nascent thesis proposes that facts are recalled from the training corpus through the MLP weights in a manner resembling key-value memory, implying in effect that "knowledge" is stored in the network. Furthermore, by modifying the MLP modules, one can control the language model's generation of factual information. The plausibility of the KN thesis has been demonstrated by the success of KN-inspired model editing methods (Dai et al., 2022; Meng et al., 2022). We find that this thesis is, at best, an oversimplification. Not only have we found that we can edit the expression of certain linguistic phenomena using the same model editing methods but, through a more comprehensive evaluation, we have found that the KN thesis does not adequately explain the process of factual expression. While it is possible to argue that the MLP weights store complex patterns that are interpretable both syntactically and semantically, these patterns do not constitute "knowledge." To gain a more comprehensive understanding of the knowledge representation process, we must look beyond the MLP weights and explore recent models' complex layer structures and attention mechanisms.