Training Language Models to Explain Their Own Computations

TL;DR

This paper demonstrates that language models can be trained to generate natural language explanations of their internal computations, leveraging their internal access to improve interpretability and generalize to new queries.

Contribution

The authors introduce a method for fine-tuning language models to produce explanations of their internal features, causal structures, and token influences, showing improved interpretability.

Findings

Explainer models generalize well to new queries with limited training data.

Models explain their own computations more effectively than different models.

Generated explanations serve as a scalable interpretability tool.

Abstract

Can language models (LMs) learn to faithfully describe their internal computations? Are they better able to describe themselves than other models? We study the extent to which LMs' privileged access to their own internals can be leveraged to produce new techniques for explaining their behavior. Using existing interpretability techniques as a source of ground truth, we fine-tune LMs to generate natural language descriptions of (1) the information encoded by LM features, (2) the causal structure of LMs' internal activations, and (3) the influence of specific input tokens on LM outputs. When trained with only tens of thousands of example explanations, explainer models exhibit non-trivial generalization to new queries. This generalization appears partly attributable to explainer models' privileged access to their own internals: using a model to explain its own computations generally works better than using a *different* model to explain its computations (even if the explainer model is significantly more capable than the target). Our results suggest not only that LMs can learn to reliably explain their internal computations, but that such explanations offer a scalable complement to existing interpretability methods. Code and data at https://github.com/TransluceAI/introspective-interp