More Aligned, Less Diverse? Analyzing the Grammar and Lexicon of Two Generations of LLMs

TL;DR

This paper compares two generations of LLMs and human news texts, revealing that newer models have reduced syntactic and lexical diversity, possibly due to instruction tuning effects.

Contribution

It introduces a formal grammar-based analysis of LLM-generated texts, highlighting diversity changes across model generations and over time.

Findings

Newer LLMs show reduced syntactic diversity.

Lexical diversity is notably lower in newer models.

English news text has remained relatively stable over time.

Abstract

This study contributes to a growing line of research in comparing LLM-generated texts with human-authored text, in this case, English news text. We focus in particular on the evaluation of syntactic properties through formal grammar frameworks. Our analysis compares two generations of LLMs in the context of two human-authored English news datasets from two different years. Employing the Head-Driven Phrase Structure Grammar (HPSG) formalism, we investigate the distributions of syntactic structures and lexical types of AI-generated texts and contrast them with the corresponding distributions in the human-authored New York Times (NYT) articles. We use diversity metrics from ecology and information theory to quantify variation in grammatical constructions and lexical types. We show that English news text has changed little in the given time frame, while newer LLMs display reduced syntactic and, especially, lexical diversity compared to older, non-instruction-tuned models. These findings point to future work in studying effects of instruction tuning, which, while enhancing coherence and adherence to prompts, may narrow the expressive range of model output.