FiLMing Multimodal Sarcasm Detection with Attention

TL;DR

This paper introduces a novel multimodal sarcasm detection model that combines RoBERTa with co-attention and FiLMed ResNet blocks, significantly improving accuracy on Twitter datasets by leveraging text-image incongruity.

Contribution

The paper proposes a new architecture integrating co-attention and FiLMed ResNet to effectively fuse multimodal data for sarcasm detection, outperforming existing methods.

Findings

Achieved 6.14% higher F1 score than previous state-of-the-art.

Effectively models context incongruity between text and images.

Demonstrates the benefit of multimodal fusion in sarcasm detection.

Abstract

Sarcasm detection identifies natural language expressions whose intended meaning is different from what is implied by its surface meaning. It finds applications in many NLP tasks such as opinion mining, sentiment analysis, etc. Today, social media has given rise to an abundant amount of multimodal data where users express their opinions through text and images. Our paper aims to leverage multimodal data to improve the performance of the existing systems for sarcasm detection. So far, various approaches have been proposed that uses text and image modality and a fusion of both. We propose a novel architecture that uses the RoBERTa model with a co-attention layer on top to incorporate context incongruity between input text and image attributes. Further, we integrate feature-wise affine transformation by conditioning the input image through FiLMed ResNet blocks with the textual features using the GRU network to capture the multimodal information. The output from both the models and the CLS token from RoBERTa is concatenated and used for the final prediction. Our results demonstrate that our proposed model outperforms the existing state-of-the-art method by 6.14% F1 score on the public Twitter multimodal sarcasm detection dataset.