SAE as a Crystal Ball: Interpretable Features Predict Cross-domain Transferability of LLMs without Training

TL;DR

This paper introduces the SAE-based Transferability Score (STS), a novel metric using sparse autoencoders to predict how well large language models will transfer across domains after post-training, without requiring actual fine-tuning.

Contribution

The paper proposes STS, an interpretable autoencoder-based metric that forecasts LLM transferability across domains before training, filling a gap in understanding post-training model shifts.

Findings

STS achieves Pearson correlation > 0.7 with actual transfer performance.

It accurately predicts transferability in supervised fine-tuning scenarios.

Initial steps show potential for extending STS to reinforcement learning.

Abstract

In recent years, pre-trained large language models have achieved remarkable success across diverse tasks. Besides the pivotal role of self-supervised pre-training, their effectiveness in downstream applications also depends critically on the post-training process, which adapts models to task-specific data and objectives. However, this process inevitably introduces model shifts that can influence performance in different domains, and how such shifts transfer remains poorly understood. To open up the black box, we propose the SAE-based Transferability Score (STS), a new metric that leverages sparse autoencoders (SAEs) to forecast post-training transferability. Taking supervised fine-tuning as an example, STS identifies shifted dimensions in SAE representations and calculates their correlations with downstream domains, enabling reliable estimation of transferability \textit{before} fine-tuning. Extensive experiments across multiple models and domains show that STS accurately predicts the transferability of supervised fine-tuning, achieving Pearson correlation coefficients above 0.7 with actual performance changes. Beyond this, we take an initial step toward extending STS to reinforcement learning. We believe that STS can serve as an {\color{black} interpretable} tool for guiding post-training strategies in LLMs. Code is available at https://github.com/PKU-ML/STS.