Beyond Via: Analysis and Estimation of the Impact of Large Language Models in Academic Papers

TL;DR

This paper analyzes how large language models influence academic paper language, revealing shifts in word usage and challenges in identifying specific models, highlighting the dynamic and heterogeneous nature of LLM impact.

Contribution

It introduces a linear, interpretable method to quantify LLM effects on academic writing and demonstrates the difficulty of classifying texts by specific models due to their similarities.

Findings

Increased use of 'beyond' and 'via' in titles

Decreased use of 'the' and 'of' in abstracts

Current classifiers struggle to identify specific LLMs in texts

Abstract

Through an analysis of arXiv papers, we report several shifts in word usage that are likely driven by large language models (LLMs) but have not previously received sufficient attention, such as the increased frequency of "beyond" and "via" in titles and the decreased frequency of "the" and "of" in abstracts. Due to the similarities among different LLMs, experiments show that current classifiers struggle to accurately determine which specific model generated a given text in multi-class classification tasks. Meanwhile, variations across LLMs also result in evolving patterns of word usage in academic papers. By adopting a direct and highly interpretable linear approach and accounting for differences between models and prompts, we quantitatively assess these effects and show that real-world LLM usage is heterogeneous and dynamic.