Sarcasm in Sight and Sound: Benchmarking and Expansion to Improve Multimodal Sarcasm Detection

TL;DR

This paper benchmarks and extends the MUStARD++ multimodal sarcasm detection dataset, improving performance through advanced encoders and addressing class imbalance with a new dataset extension from House MD.

Contribution

It provides a comprehensive benchmark of state-of-the-art multimodal encoders on MUStARD++, introduces MUStARD++ Balanced to address sarcasm type imbalance, and adds diverse new data from House MD.

Findings

Achieved 2% macro-F1 improvement over existing benchmarks.

Further 2.4% macro-F1 boost with MUStARD++ Balanced.

Added diverse clips from House MD to enhance dataset diversity.

Abstract

The introduction of the MUStARD dataset, and its emotion recognition extension MUStARD++, have identified sarcasm to be a multi-modal phenomenon -- expressed not only in natural language text, but also through manners of speech (like tonality and intonation) and visual cues (facial expression). With this work, we aim to perform a rigorous benchmarking of the MUStARD++ dataset by considering state-of-the-art language, speech, and visual encoders, for fully utilizing the totality of the multi-modal richness that it has to offer, achieving a 2\% improvement in macro-F1 over the existing benchmark. Additionally, to cure the imbalance in the `sarcasm type' category in MUStARD++, we propose an extension, which we call \emph{MUStARD++ Balanced}, benchmarking the same with instances from the extension split across both train and test sets, achieving a further 2.4\% macro-F1 boost. The new clips were taken from a novel source -- the TV show, House MD, which adds to the diversity of the dataset, and were manually annotated by multiple annotators with substantial inter-annotator agreement in terms of Cohen's kappa and Krippendorf's alpha. Our code, extended data, and SOTA benchmark models are made public.