GPT Editors, Not Authors: The Stylistic Footprint of LLMs in Academic Preprints

TL;DR

This study investigates how large language models are used in academic preprints, finding that their usage is consistent and primarily for editing, which may mitigate hallucination risks in scholarly writing.

Contribution

It introduces a method to detect LLM influence in academic texts and reveals that LLMs are mainly used for editing rather than generating original content.

Findings

LLM influence is not predictive of stylistic segmentation.

Authors use LLMs uniformly, mainly for editing tasks.

Reduced risk of hallucinations in LLM-assisted academic writing.

Abstract

The proliferation of Large Language Models (LLMs) in late 2022 has impacted academic writing, threatening credibility, and causing institutional uncertainty. We seek to determine the degree to which LLMs are used to generate critical text as opposed to being used for editing, such as checking for grammar errors or inappropriate phrasing. In our study, we analyze arXiv papers for stylistic segmentation, which we measure by varying a PELT threshold against a Bayesian classifier trained on GPT-regenerated text. We find that LLM-attributed language is not predictive of stylistic segmentation, suggesting that when authors use LLMs, they do so uniformly, reducing the risk of hallucinations being introduced into academic preprints.