Assessing construct reliability through open-ended survey response analysis

TL;DR

This paper shows that using software to detect copied open-ended survey responses can help identify inattentive participants and improve survey reliability.

Contribution

The study introduces a novel method for identifying inattentive participants in online surveys using text originality checks.

Findings

18-35% of participants in four surveys were found to have copied open-ended responses.

Participants who wrote original responses showed higher indicator and internal consistency reliability.

Original responses were associated with more consistent validated scale responses, indicating greater attentiveness.

Abstract

Online surveys often include quantitative attention checks, but inattentive participants might also be identified using their qualitative responses. We used the software Turnitin™ to assess the originality of open-ended responses in four mixed-method surveys that included validated multi-item rating scales (i.e., constructs). Across surveys, 18-35% of participants (n =  3,771) were identified as having copied responses from online sources. We assessed indicator reliability and internal consistency reliability and found that both were lower for participants identified as using copied text versus those who wrote more original responses. Those who provided more original responses also provided more consistent responses to the validated scales, suggesting that these participants were more attentive. We conclude that this process can be used to screen open-ended responses from online surveys. We encourage future research to replicate this screening process using similar tools, investigate strategies to reduce copying behaviour, and explore the motivation of participants to search for information online, including what sources they find compelling.

Figures