Flee the Flaw: Annotating the Underlying Logic of Fallacious Arguments Through Templates and Slot-filling

TL;DR

This paper introduces explainable templates for identifying logical fallacies in arguments, annotates a dataset with these templates, and evaluates model performance, highlighting challenges in automatic fallacy detection.

Contribution

It presents a novel set of templates for explicating fallacious logic, an annotated dataset, and an analysis of model limitations in detecting fallacies.

Findings

High annotation agreement (Krippendorf's alpha 0.54)

Reasonable coverage of 0.83 in annotations

State-of-the-art models achieve only 0.47 accuracy in fallacy detection

Abstract

Prior research in computational argumentation has mainly focused on scoring the quality of arguments, with less attention on explicating logical errors. In this work, we introduce four sets of explainable templates for common informal logical fallacies designed to explicate a fallacy's implicit logic. Using our templates, we conduct an annotation study on top of 400 fallacious arguments taken from LOGIC dataset and achieve a high agreement score (Krippendorf's alpha of 0.54) and reasonable coverage (0.83). Finally, we conduct an experiment for detecting the structure of fallacies and discover that state-of-the-art language models struggle with detecting fallacy templates (0.47 accuracy). To facilitate research on fallacies, we make our dataset and guidelines publicly available.