What Are LLMs Doing to Scientific Communication? Measuring Changes in Writing Practices and Reading Experience

TL;DR

This study investigates how large language models influence scientific writing and reading, revealing significant stylistic changes and mixed perceptions of improved understandability versus negative attitudes.

Contribution

It provides the first comprehensive analysis of LLM impact on scientific communication, combining diachronic lexical analysis, stylistic feature modeling, and expert reading experience evaluation.

Findings

Word usage and contexts have changed over time, indicating semantic shifts.

LLM-modified texts show more complex syntax and longer words, with reduced lexical diversity.

Experts find LLM-enhanced texts more understandable and exciting, but also hold negative attitudes.

Abstract

Has the style of scientific communication changed due to the growing use of large language models in the writing process? We address this question in the domain of Natural Language Processing by leveraging two data resources we create: a naturalistic corpus of over 37,000 papers from the ACL Anthology (2020-2024); and a synthetic dataset of 3,000 human-written passages and their LLM-generated improvements. We first implement a series of diachronic lexical analyses, showing that both word frequency and usage contexts have changed significantly over time, indicating semantic specialization in some cases and generalization in others. Broadening our perspective, we then model a range of more complex stylistic features and find that LLM-modified texts more frequently contain certain syntactic constructions, more complex and longer words and a lower lexical diversity. Finally, we connect these changes in writing practices to subjective reading experience through a pilot annotation study with 20 domain experts. They overall rate LLM-improved texts as more understandable and exciting, but also express negative qualitative attitudes towards LLMs, highlighting the strongly subjective effect of AI-assisted writing on reading experience.