Neuron Empirical Gradient: Discovering and Quantifying Neurons Global Linear Controllability

TL;DR

This paper introduces the neuron empirical gradient (NEG), a method to quantify how neuron activations in language models influence outputs, enabling large-scale analysis of neuron behavior and knowledge representation.

Contribution

We propose NeurGrad for efficient computation of NEG, revealing global linear relationships between neuron activations and outputs, advancing understanding of neuron roles in language models.

Findings

NEG captures language skills across diverse prompts

NEG effectively represents model knowledge on MCEval8k

NEG-based analysis shows properties like efficiency and robustness

Abstract

While feed-forward neurons in pre-trained language models (PLMs) can encode knowledge, past research targeted a small subset of neurons that heavily influence outputs. This leaves the broader role of neuron activations unclear, limiting progress in areas like knowledge editing. We uncover a global linear relationship between neuron activations and outputs using neuron interventions on a knowledge probing dataset. The gradient of this linear relationship, which we call the neuron empirical gradient (NEG), captures how changes in activations affect predictions. To compute NEG efficiently, we propose NeurGrad, enabling large-scale analysis of neuron behavior in PLMs. We also show that NEG effectively captures language skills across diverse prompts through skill neuron probing. Experiments on MCEval8k, a multi-genre multiple-choice knowledge benchmark, support NEG's ability to represent model knowledge. Further analysis highlights the key properties of NEG-based skill representation: efficiency, robustness, flexibility, and interdependency. The code and data are released.