Knowledge Neurons in Pretrained Transformers

TL;DR

This paper investigates how factual knowledge is stored in pretrained Transformers by identifying and analyzing specific neurons responsible for facts, enabling knowledge editing without fine-tuning.

Contribution

It introduces the concept of knowledge neurons, proposes a method to identify them, and demonstrates their use in editing factual knowledge in pretrained models.

Findings

Knowledge neurons are positively correlated with fact expression.

Identified neurons can be used to update or erase specific facts.

The approach provides insights into how knowledge is stored in Transformers.

Abstract

Large-scale pretrained language models are surprisingly good at recalling factual knowledge presented in the training corpus. In this paper, we present preliminary studies on how factual knowledge is stored in pretrained Transformers by introducing the concept of knowledge neurons. Specifically, we examine the fill-in-the-blank cloze task for BERT. Given a relational fact, we propose a knowledge attribution method to identify the neurons that express the fact. We find that the activation of such knowledge neurons is positively correlated to the expression of their corresponding facts. In our case studies, we attempt to leverage knowledge neurons to edit (such as update, and erase) specific factual knowledge without fine-tuning. Our results shed light on understanding the storage of knowledge within pretrained Transformers. The code is available at https://github.com/Hunter-DDM/knowledge-neurons.