Boosting Logical Fallacy Reasoning in LLMs via Logical Structure Tree

TL;DR

This paper introduces a logical structure tree to improve logical fallacy detection in LLMs by explicitly modeling hierarchical logical relations, leading to significant performance improvements.

Contribution

The paper presents a novel unsupervised method to construct logical structure trees and integrates them into LLMs for enhanced fallacy reasoning.

Findings

Improved precision and recall in fallacy detection

Effective use of logical structure trees as prompts

Significant performance gains over baseline models

Abstract

Logical fallacy uses invalid or faulty reasoning in the construction of a statement. Despite the prevalence and harmfulness of logical fallacies, detecting and classifying logical fallacies still remains a challenging task. We observe that logical fallacies often use connective words to indicate an intended logical relation between two arguments, while the argument semantics does not actually support the logical relation. Inspired by this observation, we propose to build a logical structure tree to explicitly represent and track the hierarchical logic flow among relation connectives and their arguments in a statement. Specifically, this logical structure tree is constructed in an unsupervised manner guided by the constituency tree and a taxonomy of connectives for ten common logical relations, with relation connectives as non-terminal nodes and textual arguments as terminal nodes, and the latter are mostly elementary discourse units. We further develop two strategies to incorporate the logical structure tree into LLMs for fallacy reasoning. Firstly, we transform the tree into natural language descriptions and feed the textualized tree into LLMs as a part of the hard text prompt. Secondly, we derive a relation-aware tree embedding and insert the tree embedding into LLMs as a soft prompt. Experiments on benchmark datasets demonstrate that our approach based on logical structure tree significantly improves precision and recall for both fallacy detection and fallacy classification.