MMSD2.0: Towards a Reliable Multi-modal Sarcasm Detection System

TL;DR

This paper introduces MMSD2.0, a corrected benchmark dataset for multi-modal sarcasm detection, and proposes a multi-view CLIP framework that leverages multi-grained cues from text and images, significantly improving detection reliability.

Contribution

The paper presents MMSD2.0, a refined dataset removing biases and unreasonable samples, and introduces a novel multi-view CLIP framework for enhanced multi-modal sarcasm detection.

Findings

MMSD2.0 outperforms previous benchmarks in reliability.

Multi-view CLIP significantly surpasses previous baselines.

The approach improves sarcasm detection accuracy across modalities.

Abstract

Multi-modal sarcasm detection has attracted much recent attention. Nevertheless, the existing benchmark (MMSD) has some shortcomings that hinder the development of reliable multi-modal sarcasm detection system: (1) There are some spurious cues in MMSD, leading to the model bias learning; (2) The negative samples in MMSD are not always reasonable. To solve the aforementioned issues, we introduce MMSD2.0, a correction dataset that fixes the shortcomings of MMSD, by removing the spurious cues and re-annotating the unreasonable samples. Meanwhile, we present a novel framework called multi-view CLIP that is capable of leveraging multi-grained cues from multiple perspectives (i.e., text, image, and text-image interaction view) for multi-modal sarcasm detection. Extensive experiments show that MMSD2.0 is a valuable benchmark for building reliable multi-modal sarcasm detection systems and multi-view CLIP can significantly outperform the previous best baselines.