Predicting Triple Scoring with Crowdsourcing-specific Features - The fiddlehead Triple Scorer at WSDM Cup 2017
Masahiro Sato (Fuji Xerox Co., Ltd.)

TL;DR
This paper presents a method for predicting relevance scores between persons and professions or nationalities by incorporating crowdsourcing-specific features, improving prediction accuracy in the WSDM Cup 2017 challenge.
Contribution
It introduces crowdsourcing-specific features related to task difficulty into the scoring model, enhancing prediction performance for relevance scores.
Findings
Features related to task difficulty correlate with judgment discrepancies.
Incorporating these features improves average score difference.
Achieved 4th place in average score difference at WSDM Cup 2017.
Abstract
The Triple Scoring Task at the WSDM Cup 2017 involves the prediction of the relevance scores between persons and professions/nationalities. The ground truth of the relevance scores was obtained by counting the vote of seven crowdworkers. I confirmed that features related to task difficulty correlate with the discrepancy among crowdworkers' judgement. This means such features are useful for predicting whether a score is in the middle or not. Hence, the features were incorporated into the prediction model of the crowdsourced relevance scores. The introduced features improve the average score difference of the prediction. The final ranking of my prediction was 4th for average score difference and 12th for both accuracy and Kendall's tau.
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsMobile Crowdsensing and Crowdsourcing · Topic Modeling · Imbalanced Data Classification Techniques
