Feature Starvation as Geometric Instability in Sparse Autoencoders

TL;DR

This paper identifies feature starvation in sparse autoencoders as a fundamental geometric instability and proposes a new adaptive elastic net architecture to address it, improving interpretability and stability.

Contribution

The paper introduces AEN-SAEs, a differentiable architecture combining elastic net regularization to mitigate feature starvation in sparse autoencoders, grounded in classical sparse regression theory.

Findings

AEN-SAEs recover feature support under mild conditions.

AEN-SAEs mitigate feature starvation without heuristics.

Empirical results on LLMs show improved feature stability.

Abstract

Sparse autoencoders (SAEs) are used to disentangle the dense, polysemantic internal representations of large language models (LLMs) into interpretable, monosemantic concepts. However, standard $\ell_1$-regularized SAEs suffer from feature starvation (dead neurons) and shrinkage bias, often requiring computationally expensive heuristic resampling and nondifferentiable hard-masking methods to bypass these challenges. We argue that feature starvation is not merely an empirical artifact of poor data diversity, but a fundamental optimization-geometric pathology of overcomplete dictionaries: the $\ell_1$-induced sparse coding map is unstable and fundamentally misaligned with shallow, amortized encoders. To address this structural instability, we introduce adaptive elastic net SAEs (AEN-SAEs), a fully differentiable architecture grounded in classical sparse regression. AEN-SAEs combine an $\ell_2$ structural term that enforces strong convexity and Lipschitz stability with adaptive $\ell_1$ reweighting that eliminates shrinkage bias and suppresses spurious features, thereby jointly controlling the curvature and interaction structure of the induced polyhedral geometry. Theoretically, we show that AEN-SAEs yield a Lipschitz-continuous sparse coding map and recover the global feature support under mild assumptions. Empirically, across synthetic settings and LLMs (Pythia 70M, Llama 3.1 8B), AEN-SAEs mitigate feature starvation without auxiliary heuristics while maintaining competitive reconstruction abilities.