Eliciting Latent Predictions from Transformers with the Tuned Lens

TL;DR

This paper introduces the tuned lens, a method to analyze how transformer models refine their predictions layer by layer, improving interpretability and enabling detection of malicious inputs.

Contribution

The paper develops the tuned lens, an improved technique for decoding hidden states in transformers, providing more reliable insights into model inference processes.

Findings

Tuned lens outperforms logit lens in predictiveness and reliability.

Latent prediction trajectories can detect malicious inputs with high accuracy.

Tuned lens reveals that model features are similar to those used in its predictions.

Abstract

We analyze transformers from the perspective of iterative inference, seeking to understand how model predictions are refined layer by layer. To do so, we train an affine probe for each block in a frozen pretrained model, making it possible to decode every hidden state into a distribution over the vocabulary. Our method, the tuned lens, is a refinement of the earlier "logit lens" technique, which yielded useful insights but is often brittle. We test our method on various autoregressive language models with up to 20B parameters, showing it to be more predictive, reliable and unbiased than the logit lens. With causal experiments, we show the tuned lens uses similar features to the model itself. We also find the trajectory of latent predictions can be used to detect malicious inputs with high accuracy. All code needed to reproduce our results can be found at https://github.com/AlignmentResearch/tuned-lens.