Are LLMs Good Zero-Shot Fallacy Classifiers?

TL;DR

This paper investigates the use of Large Language Models (LLMs) for zero-shot fallacy classification, demonstrating their potential to outperform traditional models especially in out-of-distribution scenarios through innovative prompting strategies.

Contribution

It introduces diverse single-round and multi-round prompting schemes to enhance LLMs' zero-shot fallacy detection capabilities, especially for smaller models.

Findings

LLMs achieve acceptable zero-shot performance compared to full-shot baselines.

Multi-round prompting improves accuracy, notably for small LLMs.

LLMs outperform baselines in out-of-distribution and open-domain tasks.

Abstract

Fallacies are defective arguments with faulty reasoning. Detecting and classifying them is a crucial NLP task to prevent misinformation, manipulative claims, and biased decisions. However, existing fallacy classifiers are limited by the requirement for sufficient labeled data for training, which hinders their out-of-distribution (OOD) generalization abilities. In this paper, we focus on leveraging Large Language Models (LLMs) for zero-shot fallacy classification. To elicit fallacy-related knowledge and reasoning abilities of LLMs, we propose diverse single-round and multi-round prompting schemes, applying different task-specific instructions such as extraction, summarization, and Chain-of-Thought reasoning. With comprehensive experiments on benchmark datasets, we suggest that LLMs could be potential zero-shot fallacy classifiers. In general, LLMs under single-round prompting schemes have achieved acceptable zero-shot performances compared to the best full-shot baselines and can outperform them in all OOD inference scenarios and some open-domain tasks. Our novel multi-round prompting schemes can effectively bring about more improvements, especially for small LLMs. Our analysis further underlines the future research on zero-shot fallacy classification. Codes and data are available at: https://github.com/panFJCharlotte98/Fallacy_Detection.