Sparse Shift Autoencoders for Identifying Concepts from Large Language Model Activations

TL;DR

This paper introduces Sparse Shift Autoencoders (SSAEs), a novel method for interpretable and controllable concept extraction from large language model activations, ensuring identifiability with weak supervision.

Contribution

The paper proposes SSAEs that learn sparse differences between embeddings, providing a theoretically grounded approach for identifiable concept disentanglement in LLM interpretability.

Findings

SSAEs achieve identifiable concept recovery across multiple datasets

Disentanglement of activations from different LLMs demonstrated

Steering specific concepts with weak supervision is effective

Abstract

Unsupervised approaches to large language model (LLM) interpretability, such as sparse autoencoders (SAEs), offer a way to decode LLM activations into interpretable and, ideally, controllable concepts. On the one hand, these approaches alleviate the need for supervision from concept labels, paired prompts, or explicit causal models. On the other hand, without additional assumptions, SAEs are not guaranteed to be identifiable. In practice, they may learn latent dimensions that entangle multiple underlying concepts. If we use these dimensions to extract vectors for steering specific LLM behaviours, this non-identifiability might result in interventions that inadvertently affect unrelated properties. In this paper, we bring the question of identifiability to the forefront of LLM interpretability research. Specifically, we introduce Sparse Shift Autoencoders (SSAEs) which learn sparse representations of differences between embeddings rather than the embeddings themselves. Crucially, we show that SSAEs are identifiable from paired observations which differ in multiple unknown concepts, but not all. With this key identifiability result, we show that we can steer single concepts with only this weak form of supervision. Finally, we empirically demonstrate identifiable concept recovery across multiple real-world language datasets by disentangling activations from different LLMs.