Pragmatic Metacognitive Prompting Improves LLM Performance on Sarcasm Detection

TL;DR

This paper presents Pragmatic Metacognitive Prompting (PMP), a novel approach that enhances Large Language Models' sarcasm detection capabilities by incorporating pragmatic reasoning and reflection, achieving state-of-the-art results.

Contribution

The introduction of PMP, which leverages pragmatics and metacognition in prompting, represents a new method to improve LLMs' sarcasm detection performance.

Findings

PMP improves sarcasm detection accuracy on MUStARD and SemEval2018 datasets.

State-of-the-art performance achieved with GPT-4o, LLaMA-3-8B, and Claude 3.5 Sonnet.

Integrating pragmatic and metacognitive strategies significantly enhances LLMs' interpretative abilities.

Abstract

Sarcasm detection is a significant challenge in sentiment analysis due to the nuanced and context-dependent nature of verbiage. We introduce Pragmatic Metacognitive Prompting (PMP) to improve the performance of Large Language Models (LLMs) in sarcasm detection, which leverages principles from pragmatics and reflection helping LLMs interpret implied meanings, consider contextual cues, and reflect on discrepancies to identify sarcasm. Using state-of-the-art LLMs such as LLaMA-3-8B, GPT-4o, and Claude 3.5 Sonnet, PMP achieves state-of-the-art performance on GPT-4o on MUStARD and SemEval2018. This study demonstrates that integrating pragmatic reasoning and metacognitive strategies into prompting significantly enhances LLMs' ability to detect sarcasm, offering a promising direction for future research in sentiment analysis.