An Empirical Investigation of Worker Communities in TopCoder

TL;DR

This study analyzes worker communities on TopCoder using real-world data, revealing distinct groups with varying reliability and trustworthiness, which can inform better resource allocation in crowdsourcing.

Contribution

It introduces an empirical approach to identify and analyze worker communities in software crowdsourcing platforms based on network clustering and behavioral metrics.

Findings

Identified four worker communities: mixed-ranked, high-ranked, mid-ranked, low-ranked.

Low-ranked community has the highest reliability at 25%.

Mixed-ranked community contains the most trustworthy workers with 16% trustworthiness.

Abstract

Software crowdsourcing platforms employ extrinsic rewards such as rating or ranking systems to motivate workers. Such rating systems are noisy and provide limited knowledge about workers' preferences and performance. To develop better understanding of worker reliability and trustworthiness in software crowdsourcing, this paper reports an empirical study conducted on more than one year's real-world data from TopCoder, one of the leading software crowdsourcing platforms. To do so, first, we create a bipartite network of active workers based on common task registrations. Then, we use the Clauset-Newman-Moore graph clustering algorithm to identify worker clusters in the network. Finally, we conduct an empirical evaluation to measure and analyze workers' behavior per identified community in the platform by workers' rating. More specifically, workers' behavior is analyzed based on their performances in terms of reliability, trustworthiness, and success; their preferences in terms of efficiency and elasticity; and strategies in terms of comfort, confidence, and deceitfulness. The main result of this study identified four communities of active workers: mixed-ranked, high-ranked, mid-ranked, and low-ranked. This study shows that the low-ranked community associates with the highest reliable workers with an average reliability of 25%, while the mixed-ranked community contains the most trustworthy workers with average trustworthiness of 16%. Such empirical evidence is beneficial to help exploring resourcing options while understanding the relations among unknown resources to improve task success.