A Comparative Study on Collecting High-Quality Implicit Reasonings at a Large-scale

TL;DR

This paper compares different methodologies for collecting high-quality implicit reasoning warrants in arguments, demonstrating effective approaches through expert evaluation and releasing a new dataset for future research.

Contribution

It introduces novel methodologies for warrant collection, evaluates their effectiveness with experts, and provides a sizable annotated dataset for natural language understanding tasks.

Findings

Methodologies enable high-quality warrant collection

Expert evaluation confirms the effectiveness of approaches

Preliminary dataset of 6,000 warrants released

Abstract

Explicating implicit reasoning (i.e. warrants) in arguments is a long-standing challenge for natural language understanding systems. While recent approaches have focused on explicating warrants via crowdsourcing or expert annotations, the quality of warrants has been questionable due to the extreme complexity and subjectivity of the task. In this paper, we tackle the complex task of warrant explication and devise various methodologies for collecting warrants. We conduct an extensive study with trained experts to evaluate the resulting warrants of each methodology and find that our methodologies allow for high-quality warrants to be collected. We construct a preliminary dataset of 6,000 warrants annotated over 600 arguments for 3 debatable topics. To facilitate research in related downstream tasks, we release our guidelines and preliminary dataset.