Model Internal Sleuthing: Finding Lexical Identity and Inflectional Features in Modern Language Models

TL;DR

This study systematically probes 25 transformer models to understand how they encode lexical identity and inflectional features across multiple languages, revealing consistent patterns in their internal representations.

Contribution

It provides the first comprehensive analysis of how modern language models encode lexical and inflectional information across diverse languages and model sizes.

Findings

Inflectional features are linearly decodable throughout models.

Lexical identity is prominent early but weakens with depth.

Models with aggressive dimensionality compression show reduced steering effectiveness.

Abstract

Large transformer-based language models dominate modern NLP, yet our understanding of how they encode linguistic information relies primarily on studies of early models like BERT and GPT-2. We systematically probe 25 models from BERT Base to Qwen2.5-7B focusing on two linguistic properties: lexical identity and inflectional features across 6 diverse languages. We find a consistent pattern: inflectional features are linearly decodable throughout the model, while lexical identity is prominent early but increasingly weakens with depth. Further analysis of the representation geometry reveals that models with aggressive mid-layer dimensionality compression show reduced steering effectiveness in those layers, despite probe accuracy remaining high. Pretraining analysis shows that inflectional structure stabilizes early while lexical identity representations continue evolving. Taken together, our findings suggest that transformers maintain inflectional features across layers, while trading off lexical identity for compact, predictive representations. Our code is available at https://github.com/ml5885/model_internal_sleuthing