Concept Component Analysis: A Principled Approach for Concept Extraction in LLMs

TL;DR

This paper introduces Concept Component Analysis (ConCA), a theoretically grounded, unsupervised framework for extracting human-interpretable concepts from LLM representations, addressing limitations of previous methods like sparse autoencoders.

Contribution

The paper proposes ConCA, a novel principled approach based on linear unmixing and latent variable modeling for concept extraction in LLMs, with theoretical justification and practical variants.

Findings

ConCA effectively extracts meaningful concepts across multiple LLMs.

Sparse ConCA variants outperform traditional autoencoders in interpretability.

Theoretical analysis clarifies the relationship between LLM representations and human concepts.

Abstract

Developing human understandable interpretation of large language models (LLMs) becomes increasingly critical for their deployment in essential domains. Mechanistic interpretability seeks to mitigate the issues through extracts human-interpretable process and concepts from LLMs' activations. Sparse autoencoders (SAEs) have emerged as a popular approach for extracting interpretable and monosemantic concepts by decomposing the LLM internal representations into a dictionary. Despite their empirical progress, SAEs suffer from a fundamental theoretical ambiguity: the well-defined correspondence between LLM representations and human-interpretable concepts remains unclear. This lack of theoretical grounding gives rise to several methodological challenges, including difficulties in principled method design and evaluation criteria. In this work, we show that, under mild assumptions, LLM representations can be approximated as a {linear mixture} of the log-posteriors over concepts given the input context, through the lens of a latent variable model where concepts are treated as latent variables. This motivates a principled framework for concept extraction, namely Concept Component Analysis (ConCA), which aims to recover the log-posterior of each concept from LLM representations through a {unsupervised} linear unmixing process. We explore a specific variant, termed sparse ConCA, which leverages a sparsity prior to address the inherent ill-posedness of the unmixing problem. We implement 12 sparse ConCA variants and demonstrate their ability to extract meaningful concepts across multiple LLMs, offering theory-backed advantages over SAEs.