Archetypal SAE: Adaptive and Stable Dictionary Learning for Concept Extraction in Large Vision Models

TL;DR

This paper introduces Archetypal SAEs, a new stable dictionary learning method for interpretability in large vision models, addressing instability issues of traditional SAEs and improving concept extraction quality.

Contribution

We propose Archetypal SAEs (A-SAE), a novel approach that constrains dictionaries to the data convex hull, significantly enhancing stability and interpretability in large-scale vision models.

Findings

RA-SAEs improve dictionary stability and interpretability

New benchmarks assess plausibility and identifiability of concepts

RA-SAEs uncover semantically meaningful concepts in vision models

Abstract

Sparse Autoencoders (SAEs) have emerged as a powerful framework for machine learning interpretability, enabling the unsupervised decomposition of model representations into a dictionary of abstract, human-interpretable concepts. However, we reveal a fundamental limitation: existing SAEs exhibit severe instability, as identical models trained on similar datasets can produce sharply different dictionaries, undermining their reliability as an interpretability tool. To address this issue, we draw inspiration from the Archetypal Analysis framework introduced by Cutler & Breiman (1994) and present Archetypal SAEs (A-SAE), wherein dictionary atoms are constrained to the convex hull of data. This geometric anchoring significantly enhances the stability of inferred dictionaries, and their mildly relaxed variants RA-SAEs further match state-of-the-art reconstruction abilities. To rigorously assess dictionary quality learned by SAEs, we introduce two new benchmarks that test (i) plausibility, if dictionaries recover "true" classification directions and (ii) identifiability, if dictionaries disentangle synthetic concept mixtures. Across all evaluations, RA-SAEs consistently yield more structured representations while uncovering novel, semantically meaningful concepts in large-scale vision models.