Weight-based Analysis of Detokenization in Language Models: Understanding the First Stage of Inference Without Inference

TL;DR

This paper presents a weight-based analytical approach to understand the detokenization process in GPT-2, revealing how model weights influence attention biases without performing inference.

Contribution

It introduces a novel decomposition of first-layer attention in GPT-2 that explains detokenization effects purely through weight analysis, bypassing inference-based methods.

Findings

Weight-based explanations reveal attention bias toward close tokens.

Decomposition quantifies contributions of position, token, and mixed effects.

Analysis enhances understanding of early-stage language model processing.

Abstract

According to the stages-of-inference hypothesis, early layers of language models map their subword-tokenized input, which does not necessarily correspond to a linguistically meaningful segmentation, to more meaningful representations that form the model's "inner vocabulary". Prior analysis of this detokenization stage has predominantly relied on probing and interventions such as path patching, which involve selecting particular inputs, choosing a subset of components that will be patched, and then observing changes in model behavior. Here, we show that several important aspects of the detokenization stage can be understood purely by analyzing model weights, without performing any model inference steps. Specifically, we introduce an analytical decomposition of first-layer attention in GPT-2. Our decomposition yields interpretable terms that quantify the relative contributions of position-related, token-related, and mixed effects. By focusing on terms in this decomposition, we discover weight-based explanations of attention bias toward close tokens and attention for detokenization.