StructLens: A Structural Lens for Language Models via Maximum Spanning Trees

TL;DR

StructLens is a novel framework that uses maximum spanning trees to analyze and visualize the internal structural organization of language model representations across layers and training stages.

Contribution

It introduces a holistic structural analysis method for language model representations, revealing how token relationships and organizational units evolve during pre-training.

Findings

Middle layers exhibit the strongest local-span organization.

Smaller local units are detectable earlier in training, larger units emerge later.

StructLens provides new insights into token organization in language models.

Abstract

Language exhibits inherent structures, a property that explains both language acquisition and language change. Given this characteristic, we expect language models to manifest their own internal structures as well. While interpretability research has investigated how models compute representations mechanistically through attention patterns and Sparse AutoEncoders, the organization of the resulting representations is overlooked. To address this gap, we introduce StructLens, a framework to analyze representations through a holistic structural view. StructLens constructs maximum spanning trees based on the semantic representations in residual streams, inspired by tree representation in dependency parsing, and provides summaries of token relationships in representation space. We analyze how contiguous tokens are also nearby in representation space and find that middle layers show the strongest local-span organization. Moreover, analysis of pre-training checkpoints reveals that smaller local units become detectable earlier in pre-training, and larger units later. Our findings demonstrate that StructLens provides insights into how models organize token representations across layers and training. Our code is available at https://github.com/naist-nlp/structlens.