Attention-Aware Answers of the Crowd
Jingzheng Tu, Guoxian Yu, Jun Wang, Carlotta Domeniconi and, Xiangliang Zhang

TL;DR
This paper introduces a probabilistic model that accounts for fluctuating worker attention in crowdsourcing, improving label quality estimation and task assignment by capturing attention-related variability.
Contribution
It proposes a novel attention-aware probabilistic model with Bayesian inference and EM algorithm to better estimate true labels and worker attention levels in crowdsourcing.
Findings
The model effectively captures the relationship between attention and label quality.
It improves the accuracy of aggregated labels compared to existing methods.
The approach estimates optimal task counts per worker based on attention changes.
Abstract
Crowdsourcing is a relatively economic and efficient solution to collect annotations from the crowd through online platforms. Answers collected from workers with different expertise may be noisy and unreliable, and the quality of annotated data needs to be further maintained. Various solutions have been attempted to obtain high-quality annotations. However, they all assume that workers' label quality is stable over time (always at the same level whenever they conduct the tasks). In practice, workers' attention level changes over time, and the ignorance of which can affect the reliability of the annotations. In this paper, we focus on a novel and realistic crowdsourcing scenario involving attention-aware annotations. We propose a new probabilistic model that takes into account workers' attention to estimate the label quality. Expectation propagation is adopted for efficient Bayesian…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsMobile Crowdsensing and Crowdsourcing · Data Stream Mining Techniques · Anomaly Detection Techniques and Applications
