Design and Evaluation of Camera-Centric Mobile Crowdsourcing Applications

TL;DR

This study investigates how different design choices in camera-based mobile crowdsourcing apps influence user participation, data quality, and quantity, finding that higher labeling demands do not deter contribution and can improve image retrieval performance.

Contribution

The paper introduces and evaluates three versions of a camera-centric crowdsourcing app with varying labeling efforts, providing insights into user motivation and data quality trade-offs.

Findings

Higher labeling effort does not reduce user contribution.

Users collected more images with higher labeling demands.

Additional labeled data improved image retrieval performance.

Abstract

The data that underlies automated methods in computer vision and machine learning, such as image retrieval and fine-grained recognition, often comes from crowdsourcing. In contexts that rely on the intrinsic motivation of users, we seek to understand how the application design affects a user's willingness to contribute and the quantity and quality of the data they capture. In this project, we designed three versions of a camera-based mobile crowdsourcing application, which varied in the amount of labeling effort requested of the user and conducted a user study to evaluate the trade-off between the level of user-contributed information requested and the quantity and quality of labeled images collected. The results suggest that higher levels of user labeling do not lead to reduced contribution. Users collected and annotated the most images using the application version with the highest requested level of labeling with no decrease in user satisfaction. In preliminary experiments, the additional labeled data supported increased performance on an image retrieval task.