Neuron-Level Knowledge Attribution in Large Language Models

TL;DR

This paper introduces a static neuron attribution method for large language models, outperforming existing techniques in identifying key neurons and analyzing knowledge types, thereby enhancing understanding of model mechanisms.

Contribution

A novel static method for pinpointing significant neurons and identifying query neurons, advancing interpretability of large language models.

Findings

Our method outperforms seven existing attribution techniques across three metrics.

We successfully identify 'value neurons' and 'query neurons' in LLMs.

Analysis of six knowledge types across attention and FFN layers enhances understanding of knowledge storage.

Abstract

Identifying important neurons for final predictions is essential for understanding the mechanisms of large language models. Due to computational constraints, current attribution techniques struggle to operate at neuron level. In this paper, we propose a static method for pinpointing significant neurons. Compared to seven other methods, our approach demonstrates superior performance across three metrics. Additionally, since most static methods typically only identify "value neurons" directly contributing to the final prediction, we propose a method for identifying "query neurons" which activate these "value neurons". Finally, we apply our methods to analyze six types of knowledge across both attention and feed-forward network (FFN) layers. Our method and analysis are helpful for understanding the mechanisms of knowledge storage and set the stage for future research in knowledge editing. The code is available on https://github.com/zepingyu0512/neuron-attribution.