LatentQA: Teaching LLMs to Decode Activations Into Natural Language

TL;DR

LatentQA introduces a natural language decoding probe for language models, enabling detailed interpretability and control of activations, outperforming existing methods and scaling effectively with larger datasets and models.

Contribution

We develop a novel natural language decoding probe for language models, creating a dataset of activations and question-answer pairs, and demonstrate its effectiveness in interpretability and control tasks.

Findings

Outperforms existing probing baselines in reading tasks

Can steer models to exhibit unseen behaviors

Scales well with larger datasets and models

Abstract

Top-down transparency typically analyzes language model activations using probes with scalar or single-token outputs, limiting the range of behaviors that can be captured. To alleviate this issue, we develop a more expressive probe that can directly output natural language, performing LatentQA: the task of answering open-ended questions about activations. A key difficulty in developing such a probe is collecting a dataset mapping activations to natural-language descriptions. In response, we propose an approach for generating a dataset of activations and associated question-answer pairs and develop a fine-tuning method for training a decoder LLM on this dataset. We then validate our decoder's fidelity by assessing its ability to read and control model activations. First, we evaluate the decoder on a number of supervised reading tasks with a known answer, such as uncovering hidden system prompts and relational knowledge extraction, and observe that it outperforms competitive probing baselines. Second, we demonstrate that the decoder is precise enough to steer the target model to exhibit behaviors unseen during training. Finally, we show that LatentQA scales well with increasing dataset and model size.