Is ChatGPT Transforming Academics' Writing Style?

TL;DR

This study analyzes how ChatGPT influences academic writing by examining one million arXiv abstracts, revealing a significant increase in LLM-style writing, especially in computer science, with about 35% showing ChatGPT-like features.

Contribution

It introduces a statistical method to detect LLM influence in academic abstracts and quantifies the extent of ChatGPT's impact across disciplines.

Findings

Approximately 35% of computer science abstracts show LLM-style features.

The model accurately distinguishes between real and ChatGPT-modified abstracts.

LLMs are increasingly shaping academic writing styles in recent years.

Abstract

Based on one million arXiv papers submitted from May 2018 to January 2024, we assess the textual density of ChatGPT's writing style in their abstracts through a statistical analysis of word frequency changes. Our model is calibrated and validated on a mixture of real abstracts and ChatGPT-modified abstracts (simulated data) after a careful noise analysis. The words used for estimation are not fixed but adaptive, including those with decreasing frequency. We find that large language models (LLMs), represented by ChatGPT, are having an increasing impact on arXiv abstracts, especially in the field of computer science, where the fraction of LLM-style abstracts is estimated to be approximately 35%, if we take the responses of GPT-3.5 to one simple prompt, "revise the following sentences", as a baseline. We conclude with an analysis of both positive and negative aspects of the penetration of LLMs into academics' writing style.