LLM or Human? Perceptions of Trust and Information Quality in Research Summaries

TL;DR

This study explores how readers perceive trust and quality in research summaries generated or edited by LLMs, revealing that perceptions significantly influence evaluations and that readers often cannot distinguish AI-generated content.

Contribution

It provides empirical insights into reader perceptions of LLM-generated scientific abstracts, highlighting the impact of beliefs on trust and quality assessments.

Findings

Participants struggle to reliably identify LLM-generated abstracts.

Beliefs about LLM involvement influence quality and trust ratings.

Three reader orientations toward LLM-assisted writing are identified.

Abstract

Large Language Models (LLMs) are increasingly used to generate and edit scientific abstracts, yet their integration into academic writing raises questions about trust, quality, and disclosure. Despite growing adoption, little is known about how readers perceive LLM-generated summaries and how these perceptions influence evaluations of scientific work. This paper presents a mixed-methods survey experiment investigating whether readers with ML expertise can distinguish between human- and LLM-generated abstracts, how actual and perceived LLM involvement affects judgments of quality and trustworthiness, and what orientations readers adopt toward AI-assisted writing. Our findings show that participants struggle to reliably identify LLM-generated content, yet their beliefs about LLM involvement significantly shape their evaluations. Notably, abstracts edited by LLMs are rated more favorably than those written solely by humans or LLMs. We also identify three distinct reader orientations toward LLM-assisted writing, offering insights into evolving norms and informing policy around disclosure and acceptable use in scientific communication.