Extracting Paragraphs from LLM Token Activations

TL;DR

This paper explores how large language models encode paragraph-level information in token activations, revealing their ability to plan ahead by analyzing and manipulating specific token signals.

Contribution

It introduces a method to extract paragraph context from token activations, enhancing understanding of LLMs' internal planning mechanisms.

Findings

Single-token activations encode paragraph-level context

Patching activations transfers significant contextual information

Insights into LLMs' capacity for planning ahead

Abstract

Generative large language models (LLMs) excel in natural language processing tasks, yet their inner workings remain underexplored beyond token-level predictions. This study investigates the degree to which these models decide the content of a paragraph at its onset, shedding light on their contextual understanding. By examining the information encoded in single-token activations, specifically the "\textbackslash n\textbackslash n" double newline token, we demonstrate that patching these activations can transfer significant information about the context of the following paragraph, providing further insights into the model's capacity to plan ahead.