Annotating Implicit Reasoning in Arguments with Causal Links

TL;DR

This paper introduces a semi-structured template to annotate implicit reasoning in arguments through causal links, enhancing understanding of argumentative logic beyond factual knowledge, and presents a crowdsourcing process for collecting such annotations.

Contribution

It proposes a novel semi-structured annotation template for implicit argumentation knowledge based on causality, and a crowdsourcing method to collect high-quality implicit reasoning data.

Findings

High inter-annotator agreement among experts.

Crowdsourcing can effectively collect implicit reasoning, but with some challenges.

Materials are publicly released to support future research.

Abstract

Most of the existing work that focus on the identification of implicit knowledge in arguments generally represent implicit knowledge in the form of commonsense or factual knowledge. However, such knowledge is not sufficient to understand the implicit reasoning link between individual argumentative components (i.e., claim and premise). In this work, we focus on identifying the implicit knowledge in the form of argumentation knowledge which can help in understanding the reasoning link in arguments. Being inspired by the Argument from Consequences scheme, we propose a semi-structured template to represent such argumentation knowledge that explicates the implicit reasoning in arguments via causality. We create a novel two-phase annotation process with simplified guidelines and show how to collect and filter high-quality implicit reasonings via crowdsourcing. We find substantial inter-annotator agreement for quality evaluation between experts, but find evidence that casts a few questions on the feasibility of collecting high-quality semi-structured implicit reasoning through our crowdsourcing process. We release our materials(i.e., crowdsourcing guidelines and collected implicit reasonings) to facilitate further research towards the structured representation of argumentation knowledge.