SurveyMan: Programming and Automatically Debugging Surveys
Emma Tosch, Emery D. Berger

TL;DR
SurveyMan is a system that helps design, deploy, and automatically debug surveys, ensuring data quality and reducing bias through static and dynamic analysis, especially useful in uncontrolled online environments.
Contribution
It introduces a domain-specific language for survey design, static analysis for feedback, and dynamic analysis for bug detection and response quality control.
Findings
Effective in detecting survey bugs
Improves response quality in crowdsourcing
Validated through case studies on Mechanical Turk
Abstract
Surveys can be viewed as programs, complete with logic, control flow, and bugs. Word choice or the order in which questions are asked can unintentionally bias responses. Vague, confusing, or intrusive questions can cause respondents to abandon a survey. Surveys can also have runtime errors: inattentive respondents can taint results. This effect is especially problematic when deploying surveys in uncontrolled settings, such as on the web or via crowdsourcing platforms. Because the results of surveys drive business decisions and inform scientific conclusions, it is crucial to make sure they are correct. We present SurveyMan, a system for designing, deploying, and automatically debugging surveys. Survey authors write their surveys in a lightweight domain-specific language aimed at end users. SurveyMan statically analyzes the survey to provide feedback to survey authors before deployment.…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsMobile Crowdsensing and Crowdsourcing · Survey Methodology and Nonresponse · Economic and Environmental Valuation
