TruEyes: Utilizing Microtasks in Mobile Apps for Crowdsourced Labeling of Machine Learning Datasets

TL;DR

TruEyes is a mobile crowdsourcing system that distributes micro-tasks to users via app ads, maintaining label quality while providing a monetization method and user engagement alternative to traditional advertising.

Contribution

This paper introduces TruEyes, a novel mobile crowdsourcing platform that leverages micro-tasks in app ads, balancing data quality and user preferences.

Findings

Label quality is comparable to traditional crowdsourcing methods.

Most users prefer task ads over traditional ads.

The system effectively engages users in data labeling.

Abstract

The growing use of supervised machine learning in research and industry has increased the need for labeled datasets. Crowdsourcing has emerged as a popular method to create data labels. However, working on large batches of tasks leads to worker fatigue, negatively impacting labeling quality. To address this, we present TruEyes, a collaborative crowdsourcing system, enabling the distribution of micro-tasks to mobile app users. TruEyes allows machine learning practitioners to publish labeling tasks, mobile app developers to integrate task ads for monetization, and users to label data instead of watching advertisements. To evaluate the system, we conducted an experiment with N=296 participants. Our results show that the quality of the labeled data is comparable to traditional crowdsourcing approaches and most users prefer task ads over traditional ads. We discuss extensions to the system and address how mobile advertisement space can be used as a productive resource in the future.