Differentiable Faithfulness Alignment for Cross-Model Circuit Transfer

TL;DR

The paper introduces Differentiable Faithfulness Alignment (DFA), a method to transfer circuit information from smaller to larger language models using learned differentiable alignment, improving interpretability and transferability.

Contribution

DFA is a novel framework that aligns source and target model circuits through a learned mapping, enabling scalable circuit transfer without full circuit discovery on the target.

Findings

DFA performs well on Llama-3 1B to 3B transfer tasks.

Aligned circuits often match or surpass direct attribution methods.

Transfer effectiveness decreases with larger model gaps and on Qwen-2.5.

Abstract

Mechanistic interpretability has made it possible to localize circuits underlying specific behaviors in language models, but existing methods are expensive, model-specific, and difficult to scale to larger architectures. We introduce \textbf{Differentiable Faithfulness Alignment (DFA)}, a framework that transfers circuit information from a smaller source model to a larger target model through a learned differentiable alignment. DFA projects source-model node importance scores into the target model and trains this mapping with a soft faithfulness objective, avoiding full circuit discovery on the target model. We evaluate DFA on Llama-3 and Qwen-2.5 across six tasks spanning factual retrieval, multiple-choice reasoning, and arithmetic. The strongest results occur on Llama-3 $1$B$\rightarrow3$B, where aligned circuits are often competitive with direct node attribution and zero-shot transfer remains effective. Recovery weakens for larger source--target gaps and is substantially lower on Qwen-2.5, suggesting that transfer becomes harder as architectural and scaling differences increase. Overall, DFA consistently outperforms simple baselines and, in some settings, recovers target-model circuits with faithfulness comparable to or stronger than direct attribution. These results suggest that smaller models can provide useful mechanistic priors for larger ones, while highlighting both the promise and the limits of node-level cross-model circuit alignment.\footnote{Code is available at https://github.com/jasonshaoshun/dfa-circuits.