A Puzzle-Based Dataset for Natural Language Inference

TL;DR

This paper introduces a new dataset of logical puzzles in natural language, designed to evaluate natural language inference and understanding by providing annotated questions with verified answers.

Contribution

It presents a novel puzzle-based dataset with verified entailment, contradiction, and ambiguity labels, emphasizing puzzle quality for machine comprehension.

Findings

Dataset covers three domains: comparing puzzles, knights and knaves, zebra puzzles.

Questions are generated from relations and individuals in the puzzles.

Verified answers ensure reliability for natural language inference tasks.

Abstract

We provide here a dataset for tasks related to natural language understanding and natural language inference. The dataset contains logical puzzles in natural language from three domains: comparing puzzles, knighs and knaves, and zebra puzzles. Each puzzle is associated with the entire set of atomic questions that can be generated based on the relations and individuals occurring in the text. For each question we provide the correct answer: entailment, contradiction or ambiguity. The answer's correctness is verified against theorem provers. Good puzzles have two properties: (i) each piece of information is necessary and (ii) no unnecessary information is provided. These properties make puzzles interesting candidates for machine comprehension tasks.