Transcoder Adapters for Reasoning-Model Diffing

TL;DR

This paper introduces transcoder adapters to interpret and analyze the internal changes in reasoning models after fine-tuning, revealing sparse, interpretable features responsible for reasoning behaviors like hesitation tokens.

Contribution

We propose transcoder adapters as a novel method to approximate and interpret the internal computation differences in reasoning models post-fine-tuning.

Findings

Adapters accurately reflect model internal computation and predictions.

Adapters recover 50-90% of accuracy gains from reasoning fine-tuning.

Only about 8% of adapter features are related to reasoning behaviors.

Abstract

While reasoning models are increasingly ubiquitous, the effects of reasoning training on a model's internal mechanisms remain poorly understood. In this work, we introduce transcoder adapters, a technique for learning an interpretable approximation of the difference in MLP computation before and after fine-tuning. We apply transcoder adapters to characterize the differences between Qwen2.5-Math-7B and its reasoning-distilled variant, DeepSeek-R1-Distill-Qwen-7B. Learned adapters are faithful to the target model's internal computation and next-token predictions. When evaluated on reasoning benchmarks, adapters match the reasoning model's response lengths and typically recover 50-90% of the accuracy gains from reasoning fine-tuning. Adapter features are sparsely activating and interpretable. When examining adapter features, we find that only ~8% have activating examples directly related to reasoning behaviors. We deeply study one such behavior -- the production of hesitation tokens (e.g., "wait"). Using attribution graphs, we trace hesitation to only ~2.4% of adapter features (5.6k total) performing one of two functions. These features are necessary and sufficient for producing hesitation tokens; removing them reduces response length, often without affecting accuracy. Overall, our results provide insight into reasoning training and suggest transcoder adapters may be useful for studying fine-tuning more broadly.