Delving into the Utilisation of ChatGPT in Scientific Publications in Astronomy

TL;DR

This study investigates the increasing use of ChatGPT in astronomy research papers by analyzing word frequency trends in published articles, highlighting its widespread adoption and urging ethical guidelines.

Contribution

It introduces a method to detect ChatGPT's influence in scientific writing by analyzing word usage trends in astronomy publications from 2000 to 2024.

Findings

Significant increase in ChatGPT-favored words in 2024

Widespread adoption of language models in astronomy papers

Trend consistent with other scientific disciplines

Abstract

Rapid progress in the capabilities of machine learning approaches in natural language processing has culminated in the rise of large language models over the last two years. Recent works have shown unprecedented adoption of these for academic writing, especially in some fields, but their pervasiveness in astronomy has not been studied sufficiently. To remedy this, we extract words that ChatGPT uses more often than humans when generating academic text and search a total of 1 million articles for them. This way, we assess the frequency of word occurrence in published works in astronomy tracked by the NASA Astrophysics Data System since 2000. We then perform a statistical analysis of the occurrences. We identify a list of words favoured by ChatGPT and find a statistically significant increase for these words against a control group in 2024, which matches the trend in other disciplines. These results suggest a widespread adoption of these models in the writing of astronomy papers. We encourage organisations, publishers, and researchers to work together to identify ethical and pragmatic guidelines to maximise the benefits of these systems while maintaining scientific rigour.