Logical Fallacy Detection
Zhijing Jin, Abhinav Lalwani, Tejas Vaidhya, Xiaoyu Shen, Yiwen Ding,, Zhiheng Lyu, Mrinmaya Sachan, Rada Mihalcea, Bernhard Sch\"olkopf

TL;DR
This paper introduces the task of logical fallacy detection, providing new datasets and showing that structure-aware classifiers outperform large language models, highlighting challenges and applications in misinformation detection.
Contribution
The paper presents new datasets for logical fallacy detection and demonstrates that structure-aware classifiers outperform pretrained language models on this task.
Findings
Structure-aware classifiers outperform large language models by 5.46% on Logic.
Performance gap indicates the difficulty of logical fallacy detection.
Potential applications in misinformation mitigation.
Abstract
Reasoning is central to human intelligence. However, fallacious arguments are common, and some exacerbate problems such as spreading misinformation about climate change. In this paper, we propose the task of logical fallacy detection, and provide a new dataset (Logic) of logical fallacies generally found in text, together with an additional challenge set for detecting logical fallacies in climate change claims (LogicClimate). Detecting logical fallacies is a hard problem as the model must understand the underlying logical structure of the argument. We find that existing pretrained large language models perform poorly on this task. In contrast, we show that a simple structure-aware classifier outperforms the best language model by 5.46% on Logic and 4.51% on LogicClimate. We encourage future work to explore this task as (a) it can serve as a new reasoning challenge for language models,…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
