Learning to Pay Attention: Unsupervised Modeling of Attentive and Inattentive Respondents in Survey Data

TL;DR

This paper introduces an unsupervised, domain-agnostic framework for detecting inattentive survey respondents by analyzing response coherence through geometric and probabilistic models, emphasizing survey design's role in detection effectiveness.

Contribution

It presents a novel label-free method combining Autoencoders and Chow-Liu trees for inattentiveness detection, highlighting the importance of survey structure for model performance.

Findings

Detection effectiveness depends more on survey structure than model complexity.

Coherent, overlapping item batteries enable reliable inattentiveness detection.

Design principles for reliable measurement also enhance algorithmic detectability.

Abstract

The integrity of behavioral and social-science surveys depends on detecting inattentive respondents who provide random or low-effort answers. Traditional safeguards, such as attention checks, are often costly, reactive, and inconsistent. We propose a unified, label-free framework for inattentiveness detection that scores response coherence using complementary unsupervised views: geometric reconstruction (Autoencoders) and probabilistic dependency modeling (Chow-Liu trees). While we introduce a "Percentile Loss" objective to improve Autoencoder robustness against anomalies, our primary contribution is identifying the structural conditions that enable unsupervised quality control. Across nine heterogeneous real-world datasets, we find that detection effectiveness is driven less by model complexity than by survey structure: instruments with coherent, overlapping item batteries exhibit strong covariance patterns that allow even linear models to reliably separate attentive from inattentive respondents. This reveals a critical ``Psychometric-ML Alignment'': the same design principles that maximize measurement reliability (e.g., internal consistency) also maximize algorithmic detectability. The framework provides survey platforms with a scalable, domain-agnostic diagnostic tool that links data quality directly to instrument design, enabling auditing without additional respondent burden.