Ring That Bell: A Corpus and Method for Multimodal Metaphor Detection in Videos

TL;DR

This paper introduces the first multimodal metaphor corpus from videos with annotations and proposes a text-based detection method achieving 62% F1-score, exploring multimodal approaches and analyzing their limitations.

Contribution

It provides the first openly available multimodal metaphor dataset and a text-based detection method, highlighting challenges in visual cue utilization.

Findings

Text-based model achieves 62% F1-score in metaphor detection.

Multimodal methods did not outperform text-only approach.

Visual cues are subtle and challenging for current models.

Abstract

We present the first openly available multimodal metaphor annotated corpus. The corpus consists of videos including audio and subtitles that have been annotated by experts. Furthermore, we present a method for detecting metaphors in the new dataset based on the textual content of the videos. The method achieves a high F1-score (62\%) for metaphorical labels. We also experiment with other modalities and multimodal methods; however, these methods did not out-perform the text-based model. In our error analysis, we do identify that there are cases where video could help in disambiguating metaphors, however, the visual cues are too subtle for our model to capture. The data is available on Zenodo.