AdaptiveK Sparse Autoencoders: Dynamic Sparsity Allocation for Interpretable LLM Representations

TL;DR

This paper introduces AdaptiveK SAE, a dynamic sparsity autoencoder that adjusts feature allocation based on input complexity, improving interpretability and reconstruction quality in large language models.

Contribution

It presents a novel framework that dynamically allocates sparsity in autoencoders guided by semantic complexity signals from LLMs, advancing interpretability methods.

Findings

Outperforms fixed-sparsity methods on multiple metrics

Effectively encodes context complexity linearly in LLM representations

Reduces hyperparameter tuning and computational costs

Abstract

Understanding the internal representations of large language models (LLMs) remains a central challenge for interpretability research. Sparse autoencoders (SAEs) offer a promising solution by decomposing activations into interpretable features, but existing approaches rely on fixed sparsity constraints that fail to account for input complexity. We propose AdaptiveK SAE (Adaptive Top K Sparse Autoencoders), a novel framework that dynamically adjusts sparsity levels based on the semantic complexity of each input. Leveraging linear probes, we demonstrate that context complexity is linearly encoded in LLM representations, and we use this signal to guide feature allocation during training. Experiments across ten language models (from 70M to 14B parameters) demonstrate that this complexity-driven adaptation significantly outperforms fixed-sparsity approaches on reconstruction fidelity, explained variance, cosine similarity and interpretability metrics while eliminating the computational burden of extensive hyperparameter tuning.