Dual Modality-Aware Gated Prompt Tuning for Few-Shot Multimodal Sarcasm Detection

TL;DR

This paper introduces DMDP, a novel few-shot multimodal sarcasm detection framework that employs gated, modality-specific deep prompts and cross-modal alignment to improve detection accuracy with limited labeled data.

Contribution

DMDP is the first to use hierarchical, modality-specific deep prompts with cross-modal alignment for few-shot multimodal sarcasm detection, enhancing feature learning and interaction.

Findings

DMDP outperforms baseline methods in few-shot settings.

The model generalizes well across different datasets.

Hierarchical prompts improve sarcasm detection accuracy.

Abstract

The widespread use of multimodal content on social media has heightened the need for effective sarcasm detection to improve opinion mining. However, existing models rely heavily on large annotated datasets, making them less suitable for real-world scenarios where labeled data is scarce. This motivates the need to explore the problem in a few-shot setting. To this end, we introduce DMDP (Deep Modality-Disentangled Prompt Tuning), a novel framework for few-shot multimodal sarcasm detection. Unlike prior methods that use shallow, unified prompts across modalities, DMDP employs gated, modality-specific deep prompts for text and visual encoders. These prompts are injected across multiple layers to enable hierarchical feature learning and better capture diverse sarcasm types. To enhance intra-modal learning, we incorporate a prompt-sharing mechanism across layers, allowing the model to aggregate both low-level and high-level semantic cues. Additionally, a cross-modal prompt alignment module enables nuanced interactions between image and text representations, improving the model's ability to detect subtle sarcastic intent. Experiments on two public datasets demonstrate DMDP's superior performance in both few-shot and extremely low-resource settings. Further cross-dataset evaluations show that DMDP generalizes well across domains, consistently outperforming baseline methods.