Daily Turking: Designing Longitudinal Daily-task Studies on Mechanical Turk

TL;DR

This paper presents a system for conducting longitudinal daily-task studies on Amazon Mechanical Turk, focusing on touch dynamics, worker engagement, and study design challenges, demonstrating its viability as an alternative to lab studies.

Contribution

It introduces a novel system for long-term daily-task studies on Mechanical Turk and evaluates key factors affecting worker retention and data quality over time.

Findings

Longitudinal studies on Mechanical Turk are feasible and effective.

Payment structure and reminders influence worker engagement.

Informed consent can be balanced with quick task completion.

Abstract

In this paper, we present our system design for conducting longitudinal daily-task studies with the same workers throughout on Amazon Mechanical Turk. We implement this system to conduct a study into touch dynamics, and present our experiences, challenges and lessons learned from doing so. Study participants installed our application on their Apple iOS phones and completed two tasks daily for 31 days. Each task involves performing a series of scrolling or swiping gestures, from which behavioral information such as movement speed or pressure is extracted. The completion of the daily tasks did not require extra interaction with the Mechanical Turk platform, yet paid workers through it. This differs somewhat from the typical rapid completion of one-off tasks that workers are used to on Amazon Mechanical Turk. This atypical use of the platform prompted us to evaluate aspects related to long-term worker retention and engagement over the study period, in particular the impacts of payment schedule (amount and structure over time) and reminder notifications. We also investigate the specific concern of reconciling informed consent with workers' desire to complete tasks quickly. We find that using the Mechanical Turk platform for conducting longitudinal daily task studies is a viable method to augment or replace traditional lab studies.