Automated Interpretability Metrics Do Not Distinguish Trained and Random Transformers

TL;DR

This paper demonstrates that common interpretability metrics for transformers cannot reliably distinguish between trained and randomly initialized models, questioning their effectiveness in measuring true learned features.

Contribution

It shows that standard SAE-based interpretability metrics are insufficient and often similar for both trained and random transformers, highlighting the need for better evaluation methods.

Findings

SAEs trained on random models yield similar interpretability scores as trained models.

High interpretability scores do not necessarily indicate learned, meaningful features.

Routine use of randomized baselines is recommended for more reliable interpretability assessment.

Abstract

Sparse autoencoders (SAEs) are widely used to extract sparse, interpretable latents from transformer activations. We test whether commonly used SAE quality metrics and automatic explanation pipelines can distinguish trained transformers from randomly initialized ones (e.g., where parameters are sampled i.i.d. from a Gaussian). Over a wide range of Pythia model sizes and multiple randomization schemes, we find that, in many settings, SAEs trained on randomly initialized transformers produce auto-interpretability scores and reconstruction metrics that are similar to those from trained models. These results show that high aggregate auto-interpretability scores do not, by themselves, guarantee that learned, computationally relevant features have been recovered. We therefore recommend treating common SAE metrics as useful but insufficient proxies for mechanistic interpretability and argue for routine randomized baselines and targeted measures of feature 'abstractness'.