Time Course MechInterp: Analyzing the Evolution of Components and Knowledge in Large Language Models

TL;DR

This paper investigates how large language models develop and organize factual knowledge during training, revealing the roles and stability of different components and the influence of task complexity.

Contribution

It introduces a detailed analysis of component roles and their evolution in LLMs, providing mechanistic insights into knowledge acquisition and representation.

Findings

Attention heads show high turnover during training.

FFNs are more stable than attention heads.

Location-based relations are learned earlier than name-based relations.

Abstract

Understanding how large language models (LLMs) acquire and store factual knowledge is crucial for enhancing their interpretability and reliability. In this work, we analyze the evolution of factual knowledge representation in the OLMo-7B model by tracking the roles of its attention heads and feed forward networks (FFNs) over the course of pre-training. We classify these components into four roles: general, entity, relation-answer, and fact-answer specific, and examine their stability and transitions. Our results show that LLMs initially depend on broad, general-purpose components, which later specialize as training progresses. Once the model reliably predicts answers, some components are repurposed, suggesting an adaptive learning process. Notably, attention heads display the highest turnover. We also present evidence that FFNs remain more stable throughout training. Furthermore, our probing experiments reveal that location-based relations converge to high accuracy earlier in training than name-based relations, highlighting how task complexity shapes acquisition dynamics. These insights offer a mechanistic view of knowledge formation in LLMs.