When Models Manipulate Manifolds: The Geometry of a Counting Task

TL;DR

This paper investigates how language models, specifically Claude 3.5 Haiku, internally represent and manipulate visual text properties like linebreaking through geometric transformations of low-dimensional manifolds, revealing complex interpretability mechanisms.

Contribution

It uncovers the geometric and feature-based processes underlying linebreaking in language models, highlighting the role of low-dimensional manifolds and attention in decision-making.

Findings

Character counts are represented on low-dimensional curved manifolds.

Attention heads twist manifolds to estimate distances for linebreaking.

Visual illusions can hijack the counting mechanism in models.

Abstract

Language models can perceive visual properties of text despite receiving only sequences of tokens-we mechanistically investigate how Claude 3.5 Haiku accomplishes one such task: linebreaking in fixed-width text. We find that character counts are represented on low-dimensional curved manifolds discretized by sparse feature families, analogous to biological place cells. Accurate predictions emerge from a sequence of geometric transformations: token lengths are accumulated into character count manifolds, attention heads twist these manifolds to estimate distance to the line boundary, and the decision to break the line is enabled by arranging estimates orthogonally to create a linear decision boundary. We validate our findings through causal interventions and discover visual illusions--character sequences that hijack the counting mechanism. Our work demonstrates the rich sensory processing of early layers, the intricacy of attention algorithms, and the importance of combining feature-based and geometric views of interpretability.