Tracing the complexity profiles of different linguistic phenomena through the intrinsic dimension of LLM representations

TL;DR

This paper investigates how the intrinsic dimension of language model representations correlates with linguistic complexity, revealing consistent patterns across models and phenomena.

Contribution

It demonstrates that intrinsic dimension differences reflect linguistic complexity contrasts and vary across layers, providing a new marker for analyzing LLMs.

Findings

ID differences align with known linguistic complexity contrasts

ID peaks occur at different layers for different phenomena

Representational similarity and pruning validate ID trends

Abstract

We explore intrinsic dimension (ID) of LLM representations as a marker of linguistic complexity. Specifically, we test whether ID differences across model layers reflect well-known complexity contrasts established in (psycho)linguistics: coordination vs. subordination, right-branching vs. center-embedding, and unambiguous vs. ambiguous attachment. Our results on six different LLMs show that these contrasts are consistently reflected in ID differences, with more complex phenomena eliciting higher ID profiles. Notably, ID differences emerge at different points across layers for different contrasts, also reaching their peaks at different stages. Further experiments using representational similarity and layer pruning confirm the trends. We conclude that ID is a useful marker of linguistic complexity in LLMs, that it points to similar linguistic processing steps across disparate LLMs, and that it has the potential to differentiate between different types of complexity.