Universal Neurons in GPT-2: Emergence, Persistence, and Functional Impact

TL;DR

This study explores the emergence, stability, and functional significance of universal neurons in GPT-2 models, revealing their consistent presence across models and training stages, and their impact on model predictions.

Contribution

It provides the first comprehensive analysis of universal neurons in GPT-2, demonstrating their emergence, persistence, and functional importance during training.

Findings

Universal neurons are consistently correlated across models.

Ablation of universal neurons significantly affects predictions.

Universal neurons are highly stable across training checkpoints.

Abstract

We investigate the phenomenon of neuron universality in independently trained GPT-2 Small models, examining these universal neurons-neurons with consistently correlated activations across models-emerge and evolve throughout training. By analyzing five GPT-2 models at five checkpoints, we identify universal neurons through pairwise correlation analysis of activations over a dataset of 5 million tokens. Ablation experiments reveal significant functional impacts of universal neurons on model predictions, measured via cross entropy loss. Additionally, we quantify neuron persistence, demonstrating high stability of universal neurons across training checkpoints, particularly in early and deeper layers. These findings suggest stable and universal representational structures emerge during language model training.