Accepted with Minor Revisions: Value of AI-Assisted Scientific Writing

TL;DR

This study investigates how large language models can assist in scientific abstract writing, revealing that AI-generated abstracts can achieve acceptance levels similar to human-written ones with minimal edits, influenced by source disclosure and perceptions.

Contribution

It introduces a novel experimental framework to evaluate AI assistance in scientific writing, highlighting the impact of source disclosure and human perception on editing and acceptance.

Findings

Authors edit more human-written abstracts than AI-generated ones without source info.

Source disclosure reduces editing differences between AI and human abstracts.

AI-generated abstracts can reach acceptance levels comparable to human ones with minimal revisions.

Abstract

Large Language Models have seen expanding application across domains, yet their effectiveness as assistive tools for scientific writing - an endeavor requiring precision, multimodal synthesis, and domain expertise - remains insufficiently understood. We examine the potential of LLMs to support domain experts in scientific writing, with a focus on abstract composition. We design an incentivized randomized controlled trial with a hypothetical conference setup where participants with relevant expertise are split into an author and reviewer pool. Inspired by methods in behavioral science, our novel incentive structure encourages authors to edit the provided abstracts to an acceptable quality for a peer-reviewed submission. Our 2 x 2 between-subject design expands into two dimensions: the implicit source of the provided abstract and the disclosure of it. We find authors make most edits when editing human-written abstracts compared to AI-generated abstracts without source attribution, often guided by higher perceived readability in AI generation. Upon disclosure of source information, the volume of edits converges in both source treatments. Reviewer decisions remain unaffected by the source of the abstract, but bear a significant correlation with the number of edits made. Careful stylistic edits, especially in the case of AI-generated abstracts, in the presence of source information, improve the chance of acceptance. We find that AI-generated abstracts hold potential to reach comparable levels of acceptability to human-written ones with minimal revision, and that perceptions of AI authorship, rather than objective quality, drive much of the observed editing behavior. Our findings reverberate the significance of source disclosure in collaborative scientific writing.