When Respondents Don't Care Anymore: Identifying the Onset of Careless Responding

TL;DR

This paper introduces a flexible machine learning method to detect the point at which respondents start answering carelessly in lengthy surveys, improving data validity and offering a software tool for researchers.

Contribution

A novel changepoint detection method for identifying onset of careless responding in surveys, with statistical guarantees and open source implementation.

Findings

High accuracy in simulation experiments

Effective discrimination between attentive and careless responses

Reveals insights into partial carelessness behaviors

Abstract

Questionnaires in the behavioral sciences tend to be lengthy. However, literature suggests that survey length is a contributing factor to careless responding, with longer questionnaires yielding higher probability that participants start responding carelessly. Consequently, in long surveys a large number of participants may engage in careless responding, posing a major threat to internal validity. We propose a novel method for identifying the onset of careless responding (or an absence thereof) that searches for a changepoint in combined measurements of multiple dimensions in which carelessness may manifest, such as inconsistency and invariability. It is highly flexible, based on machine learning, and provides statistical guarantees for controlling the false positive rate. In simulation experiments, the proposed method achieves high accuracy in identifying carelessness onset and discriminates well between attentive and various types of careless responding, even when a large number of careless respondents are present. An empirical application highlights how identifying partial carelessness uncovers novel insights on careless responding behavior. Furthermore, we provide the freely available open source software package "carelessonset" to facilitate adoption by empirical researchers.