Crowd Guilds: Worker-led Reputation and Feedback on Crowdsourcing Platforms
Mark E. Whiting, Dilrukshi Gamage, Snehalkumar S. Gaikwad, Aaron, Gilbee, Shirish Goyal, Alipta Ballav, Dinesh Majeti, Nalin Chhibber, Angela, Richmond-Fuller, Freddie Vargus, Tejas Seshadri Sarma, Varshine, Chandrakanthan, Teogenes Moura, Mohamed Hashim Salih

TL;DR
This paper introduces crowd guilds, worker-led groups that improve reputation accuracy on crowdsourcing platforms by using peer assessment, demonstrating better correlation with actual worker quality than traditional decentralized models.
Contribution
It proposes a novel crowd guild model inspired by historical worker guilds, implementing peer assessment to enhance reputation accuracy in crowdsourcing.
Findings
Crowd guilds produce reputation signals more correlated with ground-truth quality.
Crowd guilds outperform traditional decentralized models in accuracy.
The field experiment shows improved reputation reliability with crowd guilds.
Abstract
Crowd workers are distributed and decentralized. While decentralization is designed to utilize independent judgment to promote high-quality results, it paradoxically undercuts behaviors and institutions that are critical to high-quality work. Reputation is one central example: crowdsourcing systems depend on reputation scores from decentralized workers and requesters, but these scores are notoriously inflated and uninformative. In this paper, we draw inspiration from historical worker guilds (e.g., in the silk trade) to design and implement crowd guilds: centralized groups of crowd workers who collectively certify each other's quality through double-blind peer assessment. A two-week field experiment compared crowd guilds to a traditional decentralized crowd work model. Crowd guilds produced reputation signals more strongly correlated with ground-truth worker quality than signals…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsMobile Crowdsensing and Crowdsourcing · Auction Theory and Applications · Open Source Software Innovations
