Causally Grounded Mechanistic Interpretability for LLMs with Faithful Natural-Language Explanations

TL;DR

This paper develops a pipeline to generate human-understandable natural language explanations for LLM internal mechanisms, using causally important attention heads and evaluating faithfulness with adapted metrics, demonstrated on GPT-2's IOI task.

Contribution

It introduces a novel method combining circuit analysis and natural language explanations, with a new faithfulness evaluation approach for mechanistic interpretability.

Findings

Identified six attention heads accounting for 61.4% of logit difference.

Circuit-based explanations achieve 100% sufficiency but only 22% comprehensiveness.

LLM-generated explanations outperform template baselines by 64% on quality metrics.

Abstract

Mechanistic interpretability identifies internal circuits responsible for model behaviors, yet translating these findings into human-understandable explanations remains an open problem. We present a pipeline that bridges circuit-level analysis and natural language explanations by (i) identifying causally important attention heads via activation patching, (ii) generating explanations using both template-based and LLM-based methods, and (iii) evaluating faithfulness using ERASER-style metrics adapted for circuit-level attribution. We evaluate on the Indirect Object Identification (IOI) task in GPT-2 Small (124M parameters), identifying six attention heads accounting for 61.4% of the logit difference. Our circuit-based explanations achieve 100% sufficiency but only 22% comprehensiveness, revealing distributed backup mechanisms. LLM-generated explanations outperform template baselines by 64% on quality metrics. We find no correlation (r = 0.009) between model confidence and explanation faithfulness, and identify three failure categories explaining when explanations diverge from mechanisms.