Is Sarcasm Detection A Step-by-Step Reasoning Process in Large Language Models?

TL;DR

This paper investigates whether sarcasm detection in large language models benefits from step-by-step reasoning or holistic cues, introducing a new prompting framework and demonstrating its effectiveness across multiple benchmarks.

Contribution

The paper proposes SarcasmCue, a novel prompting framework with four methods, and provides comprehensive empirical evidence showing its superiority in sarcasm detection across various models and datasets.

Findings

CoC and GoC perform best with GPT-4 and Claude 3.5.

ToC significantly improves performance on smaller LLMs.

Framework achieves state-of-the-art results, improving F1 scores substantially.

Abstract

Elaborating a series of intermediate reasoning steps significantly improves the ability of large language models (LLMs) to solve complex problems, as such steps would evoke LLMs to think sequentially. However, human sarcasm understanding is often considered an intuitive and holistic cognitive process, in which various linguistic, contextual, and emotional cues are integrated to form a comprehensive understanding, in a way that does not necessarily follow a step-by-step fashion. To verify the validity of this argument, we introduce a new prompting framework (called SarcasmCue) containing four sub-methods, viz. chain of contradiction (CoC), graph of cues (GoC), bagging of cues (BoC) and tensor of cues (ToC), which elicits LLMs to detect human sarcasm by considering sequential and non-sequential prompting methods. Through a comprehensive empirical comparison on four benchmarks, we highlight three key findings: (1) CoC and GoC show superior performance with more advanced models like GPT-4 and Claude 3.5, with an improvement of 3.5%. (2) ToC significantly outperforms other methods when smaller LLMs are evaluated, boosting the F1 score by 29.7% over the best baseline. (3) Our proposed framework consistently pushes the state-of-the-art (i.e., ToT) by 4.2%, 2.0%, 29.7%, and 58.2% in F1 scores across four datasets. This demonstrates the effectiveness and stability of the proposed framework.