Poster Session I - A52 THE USE OF LARGE LANGUAGE MODELS IN GASTROENTEROLOGY LITERATURE: A GROWING ARTIFICIAL INTELLIGENCE FOOTPRINT

TL;DR

This study shows a sharp rise in AI-generated language in gastroenterology research abstracts since the introduction of ChatGPT, with higher use in top and lower-impact journals.

Contribution

The study quantifies the adoption of LLMs in gastroenterology literature and reveals a U-shaped pattern across journal impact factors.

Findings

AI-like language in GI abstracts rose from 0.07% in 2016 to 4.68% in 2024.

Adoption was highest in Q1 and Q4 journals and lowest in Q2/Q3 journals.

Lexical diversity remained stable despite the increase in AI-related language.

Abstract

The integration of Large Language Models (LLMs) into academic writing has transformed scientific literature, but the adoption within the gastrointestinal (GI) literature remains to be quantified.

In this study, we aim to estimate the proportion of classical LLM-related language in GI abstracts from 2010 to 2024, and to characterize its variation across impact factor (IF) quartiles, and impact on lexical patterns.

We conducted a retrospective, bibliometric analysis of 158,473 PubMed-indexed GI abstracts, sourced from all GI-related journals with 2024 IF ≥ 2 as found on Clarivate. A synthetic corpus of 10,000 GPT-3.5 generated abstracts was used to model AI-like linguistic distributions. The annual proportion of AI-like text (α) was estimated using a maximum-likelihood mixture model with Laplace smoothing. Journals were stratified into quartiles by their 2024 IF for sub-analysis. Lexical diversity was quantified on a yearly basis, using the type-token ratio (TTR).

Between 2010-2019, α was negligible (<0.001%), with only a discrete inflection beginning in 2016 (α = 0.07%). A sharp rise is noted after the introduction of ChatGPT, reaching successively 0.86% (2020), 1.77% (2021), 1.35% (2022), 1.95% (2023), and finally up to 4.68% (2024). By IF quartile, α demonstrated a U-shaped curve, lowest in Q2/Q3 (4.2% and 2.97% respectively), and highest in Q1/Q4 (5.4% and 5.7%), suggesting disproportionate adoption in both high and lower-impact journals. Lexical diversity remained stable throughout a measured period of 15 years (TTR range 0.0067 to 0.0089), demonstrating that the increased AI-related language was not associated to measurable shifts in vocabulary.

The prevalence of LLM-related language in GI abstracts has increased sharply since the mass-introduction of LLMs, with a five-fold surge subsequent to the release of ChatGPT. These findings suggest a growing integration of LLMs in the GI body of knowledge, suggesting a need for clear editorial policies and standards of transparency regarding AI-assisted writing.

None

Figures