Scientific production in the era of Large Language Models

TL;DR

This study examines how Large Language Models are transforming scientific research, increasing paper output, altering quality perceptions, and changing citation behaviors across various fields based on large-scale data analysis.

Contribution

It provides a comprehensive analysis of LLM impact on scientific production, revealing significant increases in output and shifts in research and citation patterns.

Findings

Scientists using LLMs show a 23.7-89.3% increase in paper production.

LLM use has changed the link between writing complexity and perceived quality.

LLM adopters cite more diverse and less-cited prior works.

Abstract

Large Language Models (LLMs) are rapidly reshaping scientific research. We analyze these changes in multiple, large-scale datasets with 2.1M preprints, 28K peer review reports, and 246M online accesses to scientific documents. We find: 1) scientists adopting LLMs to draft manuscripts demonstrate a large increase in paper production, ranging from 23.7-89.3% depending on scientific field and author background, 2) LLM use has reversed the relationship between writing complexity and paper quality, leading to an influx of manuscripts that are linguistically complex but substantively underwhelming, and 3) LLM adopters access and cite more diverse prior work, including books and younger, less-cited documents. These findings highlight a stunning shift in scientific production that will likely require a change in how journals, funding agencies, and tenure committees evaluate scientific works.