Can Large Vision-Language Models Understand Multimodal Sarcasm?

TL;DR

This paper evaluates the capabilities of Large Visual Language Models in understanding and explaining multimodal sarcasm, identifying limitations, and proposing a training-free framework to enhance their interpretative abilities.

Contribution

It introduces a novel training-free framework that combines object extraction and external knowledge to improve LVLMs in multimodal sarcasm detection and explanation.

Findings

LVLMs have limited visual understanding for sarcasm detection.

Incorporating external knowledge improves sarcasm interpretation.

The proposed framework enhances model performance without additional training.

Abstract

Sarcasm is a complex linguistic phenomenon that involves a disparity between literal and intended meanings, making it challenging for sentiment analysis and other emotion-sensitive tasks. While traditional sarcasm detection methods primarily focus on text, recent approaches have incorporated multimodal information. However, the application of Large Visual Language Models (LVLMs) in Multimodal Sarcasm Analysis (MSA) remains underexplored. In this paper, we evaluate LVLMs in MSA tasks, specifically focusing on Multimodal Sarcasm Detection and Multimodal Sarcasm Explanation. Through comprehensive experiments, we identify key limitations, such as insufficient visual understanding and a lack of conceptual knowledge. To address these issues, we propose a training-free framework that integrates in-depth object extraction and external conceptual knowledge to improve the model's ability to interpret and explain sarcasm in multimodal contexts. The experimental results on multiple models show the effectiveness of our proposed framework. The code is available at https://github.com/cp-cp/LVLM-MSA.