Exploring the change in scientific readability following the release of ChatGPT

TL;DR

This study analyzes arXiv abstracts from 2010 to 2024 to assess changes in readability, finding a steady decline over time and significant shifts after ChatGPT's release, indicating AI's influence on scientific writing.

Contribution

It provides a comprehensive analysis of scientific abstract readability over time and highlights the impact of ChatGPT on the complexity of scientific language.

Findings

Readability of abstracts has steadily decreased over the years.

Significant changes in readability occurred after ChatGPT's release.

Most categories experienced increased complexity in 2023 and 2024.

Abstract

The rise and growing popularity of accessible large language models have raised questions about their impact on various aspects of life, including how scientists write and publish their research. The primary objective of this paper is to analyze a dataset consisting of all abstracts posted on arXiv.org between 2010 and June 7th, 2024, to assess the evolution of their readability and determine whether significant shifts occurred following the release of ChatGPT in November 2022. Four standard readability formulas are used to calculate individual readability scores for each paper, classifying their level of readability. These scores are then aggregated by year and across the eight primary categories covered by the platform. The results show a steady annual decrease in readability, suggesting that abstracts are likely becoming increasingly complex. Additionally, following the release of ChatGPT, a significant change in readability is observed for 2023 and the analyzed months of 2024. Similar trends are found across categories, with most experiencing a notable change in readability during 2023 and 2024. These findings offer insights into the broader changes in readability and point to the likely influence of AI on scientific writing.