Study on Patterns and Effect of Task Diversity in Software Crowdsourcing

TL;DR

This study empirically investigates how different patterns of task diversity in software crowdsourcing affect success metrics, revealing key attributes and configurations that optimize task completion and worker reliability.

Contribution

It introduces a conceptual model for measuring task diversity and identifies how specific diversity patterns influence platform success and worker performance.

Findings

Monetary prize and task complexity are key attributes differentiating competition levels.

Three task diversity patterns identified: responsive to prize, responsive to prize and complexity, over responsive to prize.

Responsive to prize pattern yields highest task density and worker reliability.

Abstract

Context: The success of software crowdsourcing depends on steady tasks supply and active worker pool. Existing analysis reveals an average task failure ratio of 15.7% in software crowdsourcing market. Goal: The objective of this study is to empirically investigate patterns and effect of task diversity in software crowdsourcing platform in order to improve the success and efficiency of software crowdsourcing. Method: We propose a conceptual task diversity model, and develop an approach to measuring and analyzing task diversity.More specifically, this includes grouping similar tasks, ranking them based on their competition level and identifying the dominant attributes that distinguish among these levels, and then studying the impact of task diversity on task success and worker performance in crowdsourcing platform. The empirical study is conducted on more than one year's real-world data from TopCoder, the leading software crowdsourcing platform. Results: We identified that monetary prize and task complexity are the dominant attributes that differentiate among different competition levels. Based on these dominant attributes, we found three task diversity patterns (configurations) from workers behavior perspective: responsive to prize, responsive to prize and complexity and over responsive to prize. This study supports that1) responsive to prize configuration provides highest level of task density and workers' reliability in a platform; 2) responsive to prize and complexity configuration leads to attracting high level of trustworthy workers; 3) over responsive to prize configuration results in highest task stability and the lowest failure ratio in the platform for not high similar tasks.