Activation Oracles: Training and Evaluating LLMs as General-Purpose Activation Explainers

TL;DR

This paper introduces Activation Oracles, trained to interpret LLM activations through natural language, demonstrating strong generalization and surpassing some existing interpretability methods across various tasks.

Contribution

It presents a generalist approach to understanding LLM activations using LatentQA-trained models called Activation Oracles, effective even in out-of-distribution scenarios.

Findings

Activation Oracles recover fine-tuned information not present in input text.

Adding diverse training data improves AO performance.

AO models match or outperform white-box baselines on multiple tasks.

Abstract

Large language model (LLM) activations are notoriously difficult to understand, with most existing techniques using complex, specialized methods for interpreting them. Recent work has proposed a simpler approach known as LatentQA: training LLMs to directly accept LLM activations as inputs and answer arbitrary questions about them in natural language. However, prior work has focused on narrow task settings for both training and evaluation. In this paper, we instead take a generalist perspective. We evaluate LatentQA-trained models, which we call Activation Oracles (AOs), in far out-of-distribution settings and examine how performance scales with training data diversity. We find that AOs can recover information fine-tuned into a model (e.g., biographical knowledge or malign propensities) that does not appear in the input text, despite never being trained with activations from a fine-tuned model. Our main evaluations are four downstream tasks where we can compare to prior white- and black-box techniques. We find that even narrowly-trained LatentQA models can generalize well, and that adding additional training datasets (such as classification tasks and a self-supervised context prediction task) yields consistent further improvements. Our best AOs match or exceed white-box baselines on all four tasks and the best overall baseline on 3 of 4. These results suggest that diversified training to answer natural-language queries imparts a general capability to verbalize information about LLM activations.