TL;DR
FoolMeTwice (FM2) is a large, challenging entailment dataset created through a multiplayer Wikipedia-based game that encourages adversarial examples and diverse reasoning strategies, improving data quality for entailment tasks.
Contribution
The paper introduces FM2, a novel dataset collected via gamification that produces more challenging and diverse entailment examples than existing datasets.
Findings
Gamification reduces shortcut solutions in entailment data.
Players generate diverse reasoning strategies, increasing data complexity.
Open source release of dataset and game code.
Abstract
We release FoolMeTwice (FM2 for short), a large dataset of challenging entailment pairs collected through a fun multi-player game. Gamification encourages adversarial examples, drastically lowering the number of examples that can be solved using "shortcuts" compared to other popular entailment datasets. Players are presented with two tasks. The first task asks the player to write a plausible claim based on the evidence from a Wikipedia page. The second one shows two plausible claims written by other players, one of which is false, and the goal is to identify it before the time runs out. Players "pay" to see clues retrieved from the evidence pool: the more evidence the player needs, the harder the claim. Game-play between motivated players leads to diverse strategies for crafting claims, such as temporal inference and diverting to unrelated evidence, and results in higher quality data…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
