Crowdsourcing for Usability Testing
Di Liu, Matthew Lease, Rebecca Kuipers, and Randolph Bias

TL;DR
Crowdsourcing platforms like Amazon Mechanical Turk can effectively supplement traditional usability testing by providing faster, cost-effective insights despite some limitations, as demonstrated through comparative studies.
Contribution
This paper empirically compares crowdsourced usability testing with traditional lab methods, highlighting its potential and methodological considerations.
Findings
Crowdsourcing offers quick and cost-effective usability insights.
Traditional lab testing provides more controlled and detailed data.
Crowdsourcing has limitations but is a valuable supplementary method.
Abstract
While usability evaluation is critical to designing usable websites, traditional usability testing can be both expensive and time consuming. The advent of crowdsourcing platforms such as Amazon Mechanical Turk and CrowdFlower offer an intriguing new avenue for performing remote usability testing with potentially many users, quick turn-around, and significant cost savings. To investigate the potential of such crowdsourced usability testing, we conducted two similar (though not completely parallel) usability studies which evaluated a graduate school's website: one via a traditional usability lab setting, and the other using crowdsourcing. While we find crowdsourcing exhibits some notable limitations in comparison to the traditional lab environment, its applicability and value for usability testing is clearly evidenced. We discuss both methodological differences for crowdsourced usability…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsMobile Crowdsensing and Crowdsourcing · Open Source Software Innovations · Software Engineering Research
