Why Does ChatGPT "Delve" So Much? Exploring the Sources of Lexical Overrepresentation in Large Language Models

TL;DR

This paper investigates why large language models overuse certain scientific words like "delve" in abstracts, finding that model architecture and training data are unlikely causes, and exploring the role of reinforcement learning from human feedback.

Contribution

The study develops a formal method to identify lexical overrepresentation in scientific texts and examines potential causes, including RLHF, highlighting the complexity of language change driven by LLMs.

Findings

Identified 21 words overused in scientific abstracts likely due to LLMs.

Model architecture and training data are unlikely causes of lexical overrepresentation.

Reinforcement learning from human feedback may influence overuse, but evidence is inconclusive.

Abstract

Scientific English is currently undergoing rapid change, with words like "delve," "intricate," and "underscore" appearing far more frequently than just a few years ago. It is widely assumed that scientists' use of large language models (LLMs) is responsible for such trends. We develop a formal, transferable method to characterize these linguistic changes. Application of our method yields 21 focal words whose increased occurrence in scientific abstracts is likely the result of LLM usage. We then pose "the puzzle of lexical overrepresentation": WHY are such words overused by LLMs? We fail to find evidence that lexical overrepresentation is caused by model architecture, algorithm choices, or training data. To assess whether reinforcement learning from human feedback (RLHF) contributes to the overuse of focal words, we undertake comparative model testing and conduct an exploratory online study. While the model testing is consistent with RLHF playing a role, our experimental results suggest that participants may be reacting differently to "delve" than to other focal words. With LLMs quickly becoming a driver of global language change, investigating these potential sources of lexical overrepresentation is important. We note that while insights into the workings of LLMs are within reach, a lack of transparency surrounding model development remains an obstacle to such research.