Understanding Language Model Circuits through Knowledge Editing

TL;DR

This paper investigates how knowledge is stored and structured within language model circuits, using systematic editing experiments on GPT-2 to reveal patterns and insights into model interpretability and knowledge organization.

Contribution

It provides the first systematic analysis of knowledge distribution and circuit responses in language model circuits, advancing interpretability research.

Findings

Circuits exhibit specific response patterns to editing attempts.

Knowledge is distributed across multiple network components.

Architectural analysis reveals the composition of knowledge-bearing circuits.

Abstract

Recent advances in language model interpretability have identified circuits, critical subnetworks that replicate model behaviors, yet how knowledge is structured within these crucial subnetworks remains opaque. To gain an understanding toward the knowledge in the circuits, we conduct systematic knowledge editing experiments on the circuits of the GPT-2 language model. Our analysis reveals intriguing patterns in how circuits respond to editing attempts, the extent of knowledge distribution across network components, and the architectural composition of knowledge-bearing circuits. These findings offer insights into the complex relationship between model circuits and knowledge representation, deepening the understanding of how information is organized within language models. Our findings offer novel insights into the ``meanings'' of the circuits, and introduce directions for further interpretability and safety research of language models.