Deriving Decoder-Free Sparse Autoencoders from First Principles

TL;DR

This paper develops a theoretical framework for decoder-free sparse autoencoders using log-sum-exp objectives, demonstrating how implicit EM dynamics influence learning and component interpretability.

Contribution

It introduces a novel single-layer encoder model with LSE and InfoMax regularization, deriving explicit EM-like behavior and insights into component collapse and diversity.

Findings

Gradient-responsibility identity holds exactly.

LSE alone causes collapse without volume control.

Variance and decorrelation prevent dead components and redundancy.

Abstract

Gradient descent on log-sum-exp (LSE) objectives performs implicit expectation--maximization (EM): the gradient with respect to each component output equals its responsibility. The same theory predicts collapse without volume control analogous to the log-determinant in Gaussian mixture models. We instantiate the theory in a single-layer encoder with an LSE objective and InfoMax regularization for volume control. Experiments confirm the theory's predictions. The gradient--responsibility identity holds exactly; LSE alone collapses; variance prevents dead components; decorrelation prevents redundancy. The model exhibits EM-like optimization dynamics in which lower loss does not correspond to better features and adaptive optimizers offer no advantage. The resulting decoder-free model learns interpretable mixture components, confirming that implicit EM theory can prescribe architectures.