Transcoders Find Interpretable LLM Feature Circuits

TL;DR

This paper introduces transcoders, a novel method for interpreting transformer models by approximating dense MLP layers with sparse ones, enabling clearer circuit analysis and insights into model behavior.

Contribution

The paper presents a new approach using transcoders for weights-based circuit analysis, improving interpretability of MLP sublayers in language models.

Findings

Transcoders perform comparably to SAEs in sparsity and interpretability.

Successfully trained on models up to 1.4B parameters.

Revealed new insights into the GPT2-small 'greater-than circuit'.

Abstract

A key goal in mechanistic interpretability is circuit analysis: finding sparse subgraphs of models corresponding to specific behaviors or capabilities. However, MLP sublayers make fine-grained circuit analysis on transformer-based language models difficult. In particular, interpretable features -- such as those found by sparse autoencoders (SAEs) -- are typically linear combinations of extremely many neurons, each with its own nonlinearity to account for. Circuit analysis in this setting thus either yields intractably large circuits or fails to disentangle local and global behavior. To address this we explore transcoders, which seek to faithfully approximate a densely activating MLP layer with a wider, sparsely-activating MLP layer. We introduce a novel method for using transcoders to perform weights-based circuit analysis through MLP sublayers. The resulting circuits neatly factorize into input-dependent and input-invariant terms. We then successfully train transcoders on language models with 120M, 410M, and 1.4B parameters, and find them to perform at least on par with SAEs in terms of sparsity, faithfulness, and human-interpretability. Finally, we apply transcoders to reverse-engineer unknown circuits in the model, and we obtain novel insights regarding the "greater-than circuit" in GPT2-small. Our results suggest that transcoders can prove effective in decomposing model computations involving MLPs into interpretable circuits. Code is available at https://github.com/jacobdunefsky/transcoder_circuits/.