Robust and Explainable Identification of Logical Fallacies in Natural Language Arguments

TL;DR

This paper develops a comprehensive, multi-stage evaluation framework for detecting and classifying logical fallacies in natural language arguments, combining robust, explainable methods with datasets to improve content moderation and misinformation detection.

Contribution

It introduces a three-stage evaluation framework, adapts datasets, and employs explainable, knowledge-based methods to enhance logical fallacy identification in NLP.

Findings

Methods show varying robustness across fallacy types

Explainability helps interpret detection results

Fallacy detection remains a challenging task

Abstract

The spread of misinformation, propaganda, and flawed argumentation has been amplified in the Internet era. Given the volume of data and the subtlety of identifying violations of argumentation norms, supporting information analytics tasks, like content moderation, with trustworthy methods that can identify logical fallacies is essential. In this paper, we formalize prior theoretical work on logical fallacies into a comprehensive three-stage evaluation framework of detection, coarse-grained, and fine-grained classification. We adapt existing evaluation datasets for each stage of the evaluation. We employ three families of robust and explainable methods based on prototype reasoning, instance-based reasoning, and knowledge injection. The methods combine language models with background knowledge and explainable mechanisms. Moreover, we address data sparsity with strategies for data augmentation and curriculum learning. Our three-stage framework natively consolidates prior datasets and methods from existing tasks, like propaganda detection, serving as an overarching evaluation testbed. We extensively evaluate these methods on our datasets, focusing on their robustness and explainability. Our results provide insight into the strengths and weaknesses of the methods on different components and fallacy classes, indicating that fallacy identification is a challenging task that may require specialized forms of reasoning to capture various classes. We share our open-source code and data on GitHub to support further work on logical fallacy identification.