TL;DR
News Ninja is a gamified platform that crowdsources high-quality annotations of linguistic bias in news, improving data reliability and fostering ongoing bias mitigation through engaging gameplay and education.
Contribution
It introduces a gamified approach for collecting media bias annotations that enhances data quality and supports long-term bias mitigation.
Findings
Crowdsourced datasets with News Ninja achieve higher inter-annotator agreement.
The game facilitates continuous data collection adaptable to news context.
It reduces costs and promotes bias awareness among players.
Abstract
Recent research shows that visualizing linguistic bias mitigates its negative effects. However, reliable automatic detection methods to generate such visualizations require costly, knowledge-intensive training data. To facilitate data collection for media bias datasets, we present News Ninja, a game employing data-collecting game mechanics to generate a crowdsourced dataset. Before annotating sentences, players are educated on media bias via a tutorial. Our findings show that datasets gathered with crowdsourced workers trained on News Ninja can reach significantly higher inter-annotator agreements than expert and crowdsourced datasets with similar data quality. As News Ninja encourages continuous play, it allows datasets to adapt to the reception and contextualization of news over time, presenting a promising strategy to reduce data collection expenses, educate players, and promote…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
