Have Faith in Faithfulness: Going Beyond Circuit Overlap When Finding Model Mechanisms

TL;DR

This paper introduces EAP-IG, a gradient-based method for identifying faithful circuits in language models, emphasizing faithfulness over overlap for understanding model mechanisms.

Contribution

The paper proposes EAP-IG, an improved method for finding faithful model circuits, addressing limitations of previous edge attribution techniques.

Findings

EAP-IG produces more faithful circuits than EAP.

Circuits with high node overlap can still lack faithfulness.

Faithfulness is a more reliable measure than overlap for circuit validity.

Abstract

Many recent language model (LM) interpretability studies have adopted the circuits framework, which aims to find the minimal computational subgraph, or circuit, that explains LM behavior on a given task. Most studies determine which edges belong in a LM's circuit by performing causal interventions on each edge independently, but this scales poorly with model size. Edge attribution patching (EAP), gradient-based approximation to interventions, has emerged as a scalable but imperfect solution to this problem. In this paper, we introduce a new method - EAP with integrated gradients (EAP-IG) - that aims to better maintain a core property of circuits: faithfulness. A circuit is faithful if all model edges outside the circuit can be ablated without changing the model's performance on the task; faithfulness is what justifies studying circuits, rather than the full model. Our experiments demonstrate that circuits found using EAP are less faithful than those found using EAP-IG, even though both have high node overlap with circuits found previously using causal interventions. We conclude more generally that when using circuits to compare the mechanisms models use to solve tasks, faithfulness, not overlap, is what should be measured.