Written Justifications are Key to Aggregate Crowdsourced Forecasts
Saketh Kotamraju, Eduardo Blanco

TL;DR
This paper shows that incorporating written justifications from crowdsourced forecasters improves the accuracy of aggregated forecasts, especially during most of the forecast horizon, and analyzes factors affecting justification reliability.
Contribution
It introduces the importance of modeling written justifications in forecast aggregation and provides empirical evidence of their benefits over traditional voting methods.
Findings
Justifications improve forecast accuracy during most of the forecast period.
Majority and weighted vote baselines are competitive with justification-based methods.
Unreliable justifications can be identified through error analysis.
Abstract
This paper demonstrates that aggregating crowdsourced forecasts benefits from modeling the written justifications provided by forecasters. Our experiments show that the majority and weighted vote baselines are competitive, and that the written justifications are beneficial to call a question throughout its life except in the last quarter. We also conduct an error analysis shedding light into the characteristics that make a justification unreliable.
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsMobile Crowdsensing and Crowdsourcing · Auction Theory and Applications · Experimental Behavioral Economics Studies
