An Investigation on How AI-Generated Responses Affect SoftwareEngineering Surveys

TL;DR

This paper investigates the impact of AI-generated responses on the validity of software engineering surveys, highlighting detection methods and emphasizing the need for combined verification to maintain research integrity.

Contribution

It identifies patterns of AI misuse in surveys and proposes a hybrid approach for detecting fabricated responses to ensure data authenticity.

Findings

49 responses showed signs of synthetic authorship

AI-generated responses mimic reasoning but conceal fabricated content

Combining automated and interpretive methods improves detection accuracy

Abstract

Survey research is a fundamental empirical method in software engineering, enabling the systematic collection of data on professional practices, perceptions, and experiences. However, recent advances in large language models (LLMs) have introduced new risks to survey integrity, as participants can use generative tools to fabricate or manipulate their responses. This study explores how LLMs are being misused in software engineering surveys and investigates the methodological implications of such behavior for data authenticity, validity, and research integrity. We collected data from two survey deployments conducted in 2025 through the Prolific platform and analyzed the content of participants' answers to identify irregular or falsified responses. A subset of responses suspected of being AI generated was examined through qualitative pattern inspection, narrative characterization, and automated detection using the Scribbr AI Detector. The analysis revealed recurring structural patterns in 49 survey responses indicating synthetic authorship, including repetitive sequencing, uniform phrasing, and superficial personalization. These false narratives mimicked coherent reasoning while concealing fabricated content, undermining construct, internal, and external validity. Our study identifies data authenticity as an emerging dimension of validity in software engineering surveys. We emphasize that reliable evidence now requires combining automated and interpretive verification procedures, transparent reporting, and community standards to detect and prevent AI generated responses, thereby protecting the credibility of surveys in software engineering.