Emergent Lexical Semantics in Neural Language Models: Testing Martin's Law on LLM-Generated Text

TL;DR

This paper investigates how neural language models develop lexical semantics related to Martin's Law during training, revealing a non-monotonic emergence and degradation pattern of semantic regularities across different model sizes and training checkpoints.

Contribution

It introduces a systematic methodology to evaluate the emergence of linguistic regularities in neural language models using clustering of contextual embeddings.

Findings

Martin's Law emerges around checkpoint 100.

Peak correlation (r > 0.6) occurs at checkpoint 104.

Larger models degrade more gracefully than smaller ones.

Abstract

We present the first systematic investigation of Martin's Law - the empirical relationship between word frequency and polysemy - in text generated by neural language models during training. Using DBSCAN clustering of contextualized embeddings as an operationalization of word senses, we analyze four Pythia models (70M-1B parameters) across 30 training checkpoints. Our results reveal a non-monotonic developmental trajectory: Martin's Law emerges around checkpoint 100, reaches peak correlation (r > 0.6) at checkpoint 104, then degrades by checkpoint 105. Smaller models (70M, 160M) experience catastrophic semantic collapse at late checkpoints, while larger models (410M, 1B) show graceful degradation. The frequency-specificity trade-off remains stable (r $\approx$ -0.3) across all models. These findings suggest that compliance with linguistic regularities in LLM-generated text is not monotonically increasing with training, but instead follows a balanced trajectory with an optimal semantic window. This work establishes a novel methodology for evaluating emergent linguistic structure in neural language models.