Do LLMs write like humans? Variation in grammatical and rhetorical styles

TL;DR

This study investigates whether large language models (LLMs) mimic human rhetorical and grammatical styles, revealing systematic differences that persist across models and are more pronounced in instruction-tuned versions, highlighting challenges in replicating human stylistic variation.

Contribution

The paper introduces a novel analysis of LLMs' rhetorical and grammatical styles, demonstrating measurable differences from human writing using Douglas Biber's linguistic features.

Findings

Differences in stylistic features between LLMs and humans are systematic and measurable.

Larger and instruction-tuned models exhibit more pronounced stylistic differences.

Advanced linguistic features can effectively distinguish LLM output from human writing.

Abstract

Large language models (LLMs) are capable of writing grammatical text that follows instructions, answers questions, and solves problems. As they have advanced, it has become difficult to distinguish their output from human-written text. While past research has found some differences in surface features such as word choice and punctuation, and developed classifiers to detect LLM output, none has studied the rhetorical styles of LLMs. Using several variants of Llama 3 and GPT-4o, we construct two parallel corpora of human- and LLM-written texts from common prompts. Using Douglas Biber's set of lexical, grammatical, and rhetorical features, we identify systematic differences between LLMs and humans and between different LLMs. These differences persist when moving from smaller models to larger ones, and are larger for instruction-tuned models than base models. This observation of differences demonstrates that despite their advanced abilities, LLMs struggle to match human stylistic variation. Attention to more advanced linguistic features can hence detect patterns in their behavior not previously recognized.