Unveiling Intrinsic Dimension of Texts: from Academic Abstract to Creative Story

TL;DR

This study explores the intrinsic dimension of texts, revealing its relationship with genre, linguistic features, and interpretability, and demonstrating how different text properties influence the complexity captured by language models.

Contribution

It is the first comprehensive analysis linking intrinsic dimension to interpretable text properties, genre stratification, and causal linguistic features using cross-encoder analysis and autoencoders.

Findings

ID is uncorrelated with entropy-based metrics after controlling for length

Scientific texts have low ID, while creative writing has high ID across models

Formal signals decrease ID; humanized signals increase ID, confirmed as causal effects

Abstract

Intrinsic dimension (ID) is an important tool in modern LLM analysis, informing studies of training dynamics, scaling behavior, and dataset structure, yet its textual determinants remain underexplored. We provide the first comprehensive study grounding ID in interpretable text properties through cross-encoder analysis, linguistic features, and sparse autoencoders (SAEs). In this work, we establish three key findings. First, ID is complementary to entropy-based metrics: after controlling for length, the two are uncorrelated, with ID capturing geometric complexity orthogonal to prediction quality. Second, ID exhibits robust genre stratification: scientific prose shows low ID (~8), encyclopedic content medium ID (~9), and creative/opinion writing high ID (~10.5) across all models tested. This reveals that contemporary LLMs find scientific text "representationally simple" while fiction requires additional degrees of freedom. Third, using SAEs, we identify causal features: scientific signals (formal tone, report templates, statistics) reduce ID; humanized signals (personalization, emotion, narrative) increase it. Steering experiments confirm these effects are causal. Thus, for contemporary models, scientific writing appears comparatively "easy", whereas fiction, opinion, and affect add representational degrees of freedom. Our multi-faceted analysis provides practical guidance for the proper use of ID and the sound interpretation of ID-based results.