Ballpark Crowdsourcing: The Wisdom of Rough Group Comparisons
Tom Hope, Dafna Shahaf

TL;DR
This paper introduces a novel crowdsourcing approach that leverages group-based rough estimates and machine learning to efficiently infer individual labels, especially for complex or continuous data, reducing costs and improving accuracy.
Contribution
It extends Ballpark Learning to regression, providing a convex optimization framework for group-based label inference in crowdsourcing, with robustness to outliers and practical evaluation.
Findings
Methods rival supervised models trained on true labels.
Outperforms standard label collection in accuracy and cost.
Effective for high-dimensional and continuous labels.
Abstract
Crowdsourcing has become a popular method for collecting labeled training data. However, in many practical scenarios traditional labeling can be difficult for crowdworkers (for example, if the data is high-dimensional or unintuitive, or the labels are continuous). In this work, we develop a novel model for crowdsourcing that can complement standard practices by exploiting people's intuitions about groups and relations between them. We employ a recent machine learning setting, called Ballpark Learning, that can estimate individual labels given only coarse, aggregated signal over groups of data points. To address the important case of continuous labels, we extend the Ballpark setting (which focused on classification) to regression problems. We formulate the problem as a convex optimization problem and propose fast, simple methods with an innate robustness to outliers. We evaluate our…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsMobile Crowdsensing and Crowdsourcing · Data Visualization and Analytics · Advanced Text Analysis Techniques
