ExpertLens: Activation steering features are highly interpretable

TL;DR

ExpertLens demonstrates that activation steering features in large language models are highly interpretable, aligning well with human concepts and providing a granular view of model representations.

Contribution

This paper introduces ExpertLens, a method for interpreting activation steering features in LLMs, showing their stability and alignment with human conceptual organization.

Findings

ExpertLens features are stable across models and datasets.

ExpertLens aligns closely with human behavioral data.

It outperforms word/sentence embedding-based alignment methods.

Abstract

Activation steering methods in large language models (LLMs) have emerged as an effective way to perform targeted updates to enhance generated language without requiring large amounts of adaptation data. We ask whether the features discovered by activation steering methods are interpretable. We identify neurons responsible for specific concepts (e.g., ``cat'') using the ``finding experts'' method from research on activation steering and show that the ExpertLens, i.e., inspection of these neurons provides insights about model representation. We find that ExpertLens representations are stable across models and datasets and closely align with human representations inferred from behavioral data, matching inter-human alignment levels. ExpertLens significantly outperforms the alignment captured by word/sentence embeddings. By reconstructing human concept organization through ExpertLens, we show that it enables a granular view of LLM concept representation. Our findings suggest that ExpertLens is a flexible and lightweight approach for capturing and analyzing model representations.