SurveyMan: Programming and Automatically Debugging Surveys

TL;DR

SurveyMan is a system that helps design, deploy, and automatically debug surveys, ensuring data quality and reducing bias through static and dynamic analysis, especially useful in uncontrolled online environments.

Contribution

It introduces a domain-specific language for survey design, static analysis for feedback, and dynamic analysis for bug detection and response quality control.

Findings

Effective in detecting survey bugs

Improves response quality in crowdsourcing

Validated through case studies on Mechanical Turk

Abstract

Surveys can be viewed as programs, complete with logic, control flow, and bugs. Word choice or the order in which questions are asked can unintentionally bias responses. Vague, confusing, or intrusive questions can cause respondents to abandon a survey. Surveys can also have runtime errors: inattentive respondents can taint results. This effect is especially problematic when deploying surveys in uncontrolled settings, such as on the web or via crowdsourcing platforms. Because the results of surveys drive business decisions and inform scientific conclusions, it is crucial to make sure they are correct. We present SurveyMan, a system for designing, deploying, and automatically debugging surveys. Survey authors write their surveys in a lightweight domain-specific language aimed at end users. SurveyMan statically analyzes the survey to provide feedback to survey authors before deployment. It then compiles the survey into JavaScript and deploys it either to the web or a crowdsourcing platform. SurveyMan's dynamic analyses automatically find survey bugs, and control for the quality of responses. We evaluate SurveyMan's algorithms analytically and empirically, demonstrating its effectiveness with case studies of social science surveys conducted via Amazon's Mechanical Turk.