MuSaG: A Multimodal German Sarcasm Dataset with Full-Modal Annotations

TL;DR

MuSaG is a novel German multimodal sarcasm dataset with aligned text, audio, and video annotations, designed to evaluate and improve sarcasm detection models across modalities in realistic social media scenarios.

Contribution

This paper introduces MuSaG, the first German multimodal sarcasm dataset with full-modal annotations, and benchmarks various models to identify gaps in current multimodal sarcasm detection capabilities.

Findings

Humans rely heavily on audio cues for sarcasm detection.

Models perform best on text modality, indicating a gap in multimodal understanding.

MuSaG enables evaluation of multimodal sarcasm detection in realistic settings.

Abstract

Sarcasm is a complex form of figurative language in which the intended meaning contradicts the literal one. Its prevalence in social media and popular culture poses persistent challenges for natural language understanding, sentiment analysis, and content moderation. With the emergence of multimodal large language models, sarcasm detection extends beyond text and requires integrating cues from audio and vision. We present MuSaG, the first German multimodal sarcasm detection dataset, consisting of 33 minutes of manually selected and human-annotated statements from German television shows. Each instance provides aligned text, audio, and video modalities, annotated separately by humans, enabling evaluation in unimodal and multimodal settings. We benchmark nine open-source and commercial models, spanning text, audio, vision, and multimodal architectures, and compare their performance to human annotations. Our results show that while humans rely heavily on audio in conversational settings, models perform best on text. This highlights a gap in current multimodal models and motivates the use of MuSaG for developing models better suited to realistic scenarios. We release MuSaG publicly to support future research on multimodal sarcasm detection and human-model alignment.