Measuring and Guiding Monosemanticity

TL;DR

This paper introduces a new metric for measuring feature monosemanticity in language models and proposes a method to improve interpretability and control by conditioning autoencoders on labeled concepts, demonstrating enhanced localization and disentanglement.

Contribution

It presents the Feature Monosemanticity Score (FMS) for quantifying feature monosemanticity and introduces Guided Sparse Autoencoders (G-SAE) that improve interpretability and control of LLMs.

Findings

G-SAE improves feature localization and disentanglement.

Enhanced interpretability and control with less quality loss.

Effective in toxicity, style, and privacy detection.

Abstract

There is growing interest in leveraging mechanistic interpretability and controllability to better understand and influence the internal dynamics of large language models (LLMs). However, current methods face fundamental challenges in reliably localizing and manipulating feature representations. Sparse Autoencoders (SAEs) have recently emerged as a promising direction for feature extraction at scale, yet they, too, are limited by incomplete feature isolation and unreliable monosemanticity. To systematically quantify these limitations, we introduce Feature Monosemanticity Score (FMS), a novel metric to quantify feature monosemanticity in latent representation. Building on these insights, we propose Guided Sparse Autoencoders (G-SAE), a method that conditions latent representations on labeled concepts during training. We demonstrate that reliable localization and disentanglement of target concepts within the latent space improve interpretability, detection of behavior, and control. Specifically, our evaluations on toxicity detection, writing style identification, and privacy attribute recognition show that G-SAE not only enhances monosemanticity but also enables more effective and fine-grained steering with less quality degradation. Our findings provide actionable guidelines for measuring and advancing mechanistic interpretability and control of LLMs.