Towards Multimodal Metaphor Understanding: A Chinese Dataset and Model for Metaphor Mapping Identification

TL;DR

This paper introduces a Chinese multimodal metaphor dataset and a novel CoT prompting-based model for identifying metaphor mappings, addressing gaps in non-English resources and enhancing NLP metaphor understanding.

Contribution

It provides a new Chinese multimodal metaphor dataset (CM3D) and proposes a hierarchical CoT prompting model (CPMMIM) for metaphor mapping identification, advancing NLP metaphor comprehension.

Findings

CPMMIM outperforms baseline models in accuracy

The dataset enables better understanding of Chinese multimodal metaphors

Hierarchical modeling improves interpretability of metaphor mappings

Abstract

Metaphors play a crucial role in human communication, yet their comprehension remains a significant challenge for natural language processing (NLP) due to the cognitive complexity involved. According to Conceptual Metaphor Theory (CMT), metaphors map a target domain onto a source domain, and understanding this mapping is essential for grasping the nature of metaphors. While existing NLP research has focused on tasks like metaphor detection and sentiment analysis of metaphorical expressions, there has been limited attention to the intricate process of identifying the mappings between source and target domains. Moreover, non-English multimodal metaphor resources remain largely neglected in the literature, hindering a deeper understanding of the key elements involved in metaphor interpretation. To address this gap, we developed a Chinese multimodal metaphor advertisement dataset (namely CM3D) that includes annotations of specific target and source domains. This dataset aims to foster further research into metaphor comprehension, particularly in non-English languages. Furthermore, we propose a Chain-of-Thought (CoT) Prompting-based Metaphor Mapping Identification Model (CPMMIM), which simulates the human cognitive process for identifying these mappings. Drawing inspiration from CoT reasoning and Bi-Level Optimization (BLO), we treat the task as a hierarchical identification problem, enabling more accurate and interpretable metaphor mapping. Our experimental results demonstrate the effectiveness of CPMMIM, highlighting its potential for advancing metaphor comprehension in NLP. Our dataset and code are both publicly available to encourage further advancements in this field.